Nov 03 18:37:09.113421 ip-10-0-143-45 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 18:37:09.512620 ip-10-0-143-45 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:37:09.512620 ip-10-0-143-45 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 18:37:09.512620 ip-10-0-143-45 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:37:09.512620 ip-10-0-143-45 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:37:09.516391 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516314    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.516391 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516391    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516395    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516399    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516402    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516405    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516408    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516410    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516413    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516416    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516418    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516421    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516423    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516427    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516429    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516432    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516434    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516437    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516439    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516441    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516446    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.516457 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516448    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516451    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516453    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516457    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516462    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516466    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516469    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516472    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516475    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516477    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516480    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516482    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516485    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516487    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516490    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516492    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516495    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516497    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516500    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.516919 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516503    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516505    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516508    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516511    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516513    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516516    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516519    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516522    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516524    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516527    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516529    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516532    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516534    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516537    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516540    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516543    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516545    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516548    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516550    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516553    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.517375 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516555    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516558    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516560    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516563    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516565    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516568    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516571    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516574    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516576    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516579    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516582    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516584    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516586    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516589    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516591    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516594    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516597    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516601    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516605    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.517837 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516607    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516610    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516613    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516615    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516618    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516620    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516623    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516626    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516630    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516632    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516635    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516637    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516640    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516643    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516645    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516648    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516651    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516653    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516656    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516658    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.518558 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516660    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516663    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516666    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516669    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516671    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516674    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.516676    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517485    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517492    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517495    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517498    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517501    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517504    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517507    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517509    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517512    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517514    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517517    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517519    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.519083 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517522    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517525    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517528    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517530    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517533    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517536    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517538    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517541    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517543    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517546    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517549    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517553    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517557    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517560    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517563    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517565    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517568    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517571    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517573    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.519589 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517576    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517578    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517581    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517583    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517585    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517588    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517590    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517593    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517596    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517598    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517601    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517603    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517605    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517609    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517612    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517615    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517617    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517619    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517622    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517624    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.520029 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517627    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517629    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517632    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517634    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517636    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517640    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517642    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517645    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517647    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517650    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517653    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517655    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517657    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517660    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517662    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517665    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517667    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517670    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517672    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.520539 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517674    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517677    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517680    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517682    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517685    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517688    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517690    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517693    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517696    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517698    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517701    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517703    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517705    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517708    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517710    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517713    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517715    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517717    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517720    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517722    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.520988 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517725    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517729    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517732    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517735    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517738    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517741    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517744    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517746    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517749    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517751    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517754    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517756    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517759    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517762    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517765    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.517767    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519239    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519250    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519254    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.521483 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519257    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519259    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519262    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519265    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519268    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519270    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519273    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519275    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519278    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519281    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519283    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519286    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519288    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519291    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519294    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519298    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519302    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519305    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519308    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.521930 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519310    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519313    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519316    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519318    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519321    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519323    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519326    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519328    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519331    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519333    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519339    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519343    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519346    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519348    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519351    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519354    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519356    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519359    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519361    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519364    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.522494 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519366    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519369    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519372    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519375    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519378    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519381    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519383    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519386    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519388    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519390    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519393    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519396    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519400    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519403    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519407    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519410    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519413    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519415    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519417    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.522959 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519420    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519422    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519425    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519427    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519431    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519433    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519436    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519438    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519441    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519443    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519446    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519449    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519451    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519454    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519456    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519459    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519461    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519464    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519466    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519469    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.523413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519471    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519473    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519476    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519478    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519481    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519483    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519485    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519488    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519491    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519493    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519496    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519498    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519501    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519504    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519506    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519509    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519511    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519514    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519516    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519519    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.523867 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519521    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519523    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519525    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519528    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.519530    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519622    2572 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519633    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519642    2572 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519649    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519656    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519661    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519666    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519670    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519673    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519686    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519689    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519692    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519695    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519698    2572 flags.go:64] FLAG: --cgroup-root=""
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519701    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519705    2572 flags.go:64] FLAG: --client-ca-file=""
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519707    2572 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519710    2572 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 18:37:09.524342 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519714    2572 flags.go:64] FLAG: --cluster-domain=""
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519717    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519720    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519723    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519726    2572 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519730    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519733    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519736    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519739    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519742    2572 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519744    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519747    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519750    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519753    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519757    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519760    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519763    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519766    2572 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519768    2572 flags.go:64] FLAG: --enable-server="true"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519771    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519778    2572 flags.go:64] FLAG: --event-burst="100"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519781    2572 flags.go:64] FLAG: --event-qps="50"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519784    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519787    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 18:37:09.524877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519795    2572 flags.go:64] FLAG: --eviction-hard=""
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519799    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519802    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519805    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519808    2572 flags.go:64] FLAG: --eviction-soft=""
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519812    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519814    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519817    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519820    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519823    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519826    2572 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519829    2572 flags.go:64] FLAG: --feature-gates=""
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519836    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519839    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519842    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519845    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519848    2572 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519851    2572 flags.go:64] FLAG: --help="false"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519854    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519857    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519860    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519863    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519866    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 18:37:09.525503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519869    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519872    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519875    2572 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519877    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519880    2572 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519883    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519886    2572 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519889    2572 flags.go:64] FLAG: --kube-reserved=""
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519891    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519894    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519898    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519901    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519904    2572 flags.go:64] FLAG: --lock-file=""
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519910    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519914    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519917    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519922    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519925    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519928    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519931    2572 flags.go:64] FLAG: --logging-format="text"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519934    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519937    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519940    2572 flags.go:64] FLAG: --manifest-url=""
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519943    2572 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519947    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 18:37:09.526038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519950    2572 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519954    2572 flags.go:64] FLAG: --max-pods="110"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519957    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519959    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519962    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519965    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519968    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519971    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519974    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519980    2572 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519983    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519986    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519989    2572 flags.go:64] FLAG: --pod-cidr=""
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519992    2572 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519995    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.519998    2572 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520001    2572 flags.go:64] FLAG: --port="10250"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520004    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520007    2572 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0bccba2f5c2a95c50"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520010    2572 flags.go:64] FLAG: --qos-reserved=""
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520019    2572 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520022    2572 flags.go:64] FLAG: --register-node="true"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520025    2572 flags.go:64] FLAG: --register-with-taints=""
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520028    2572 flags.go:64] FLAG: --registry-burst="10"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520031    2572 flags.go:64] FLAG: --registry-qps="5"
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520034    2572 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 18:37:09.526642 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520037    2572 flags.go:64] FLAG: --reserved-memory=""
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520041    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520044    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520047    2572 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520050    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520053    2572 flags.go:64] FLAG: --runonce="false"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520056    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520059    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520062    2572 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520065    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520068    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520071    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520074    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520077    2572 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520080    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520083    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520085    2572 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520088    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520091    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520094    2572 flags.go:64] FLAG: --system-cgroups=""
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520096    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520102    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520105    2572 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520108    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520115    2572 flags.go:64] FLAG: --tls-min-version=""
Nov 03 18:37:09.527263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520119    2572 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520124    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520127    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520130    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520133    2572 flags.go:64] FLAG: --v="2"
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520153    2572 flags.go:64] FLAG: --version="false"
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520159    2572 flags.go:64] FLAG: --vmodule=""
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520163    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520166    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520296    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520300    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520303    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520306    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520308    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520311    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520313    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520316    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520318    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520321    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520323    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520326    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.527844 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520328    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520331    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520333    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520336    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520338    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520341    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520343    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520346    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520348    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520351    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520353    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520356    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520364    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520368    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520371    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520374    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520377    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520379    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520382    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.528367 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520384    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520387    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520390    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520392    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520395    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520397    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520400    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520402    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520405    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520407    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520410    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520413    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520415    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520418    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520420    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520423    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520426    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520428    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520430    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520433    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.528826 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520435    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520438    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520440    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520443    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520445    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520450    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520454    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520456    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520459    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520461    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520464    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520466    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520469    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520471    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520474    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520476    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520479    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520481    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520484    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.529346 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520486    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520489    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520491    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520494    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520496    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520499    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520501    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520503    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520506    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520508    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520511    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520513    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520515    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520518    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520520    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520523    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520525    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520528    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520532    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520536    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.529784 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520540    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520543    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520546    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520549    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520552    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520555    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520557    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520559    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520562    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520564    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520567    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520569    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520572    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520574    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520577    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520579    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.530268 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.520607    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520642    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520646    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520649    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520652    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520654    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520656    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520659    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520662    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520664    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520667    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520669    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520672    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520676    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520681    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520686    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520689    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520692    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520695    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.530640 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520698    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520700    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520703    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520706    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520708    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520711    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520714    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520716    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520719    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520721    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520724    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520726    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520729    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520731    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520734    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520736    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520739    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520742    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520744    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520755    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.531090 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520758    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520760    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520763    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520766    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520768    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520771    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520773    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520777    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520781    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520783    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520786    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520789    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520792    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520794    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520798    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520800    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520803    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520805    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520808    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.531583 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520811    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520814    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520816    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520819    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520821    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520824    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520826    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520829    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520832    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520834    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520836    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520839    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520841    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520844    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520846    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520849    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520851    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520854    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520856    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520858    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.532034 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520863    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520868    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520871    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520874    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520876    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520879    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520883    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520885    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520888    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520890    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520893    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520895    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520898    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520900    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520903    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520905    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520907    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520910    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520912    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.532534 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520916    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520918    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520921    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520923    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520926    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520928    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520931    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520933    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520935    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.520938    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.527964    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.527978    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.527990    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.527995    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528050    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528054    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528058    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528061    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528065    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528068    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.532989 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528071    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528074    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528077    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528081    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528084    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528087    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528089    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528092    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528095    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528097    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528100    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528102    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528105    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528107    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528109    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528112    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528114    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528117    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528119    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.533459 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528122    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528124    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528127    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528130    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528133    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528135    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528150    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528153    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528156    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528159    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528162    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528165    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528168    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528171    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528173    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528176    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528178    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528181    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528183    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528186    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.533897 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528188    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528190    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528193    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528195    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528198    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528200    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528203    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528205    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528207    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528210    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528213    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528215    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528218    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528220    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528223    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528226    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528228    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528231    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528233    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528236    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.534502 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528238    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528241    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528243    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528246    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528248    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528251    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528254    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528256    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528258    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528261    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528263    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528266    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528268    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528271    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528273    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528275    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528278    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528280    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528283    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528285    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.534968 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528287    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528290    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528292    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528295    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528298    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528302    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528305    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528308    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528311    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528313    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528316    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528318    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528321    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528324    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528326    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528329    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528331    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528333    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528336    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.535445 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528339    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528341    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.528369    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528507    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528512    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528515    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528518    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528520    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528523    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528525    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528527    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528530    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528532    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528535    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528537    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528540    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.535884 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528543    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528545    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528548    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528550    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528553    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528556    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528558    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528561    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528564    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528566    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528569    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528571    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528574    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528576    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528579    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528581    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528587    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528590    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528592    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.536265 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528595    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528597    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528600    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528602    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528605    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528608    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528630    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528634    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528637    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528640    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528643    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528645    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528648    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528650    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528653    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528655    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528658    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528661    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528663    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.536718 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528666    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528668    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528672    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528676    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528678    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528681    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528683    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528686    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528688    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528691    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528693    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528696    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528699    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528701    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528704    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528706    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528709    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528711    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528714    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.537184 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528716    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528718    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528721    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528723    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528726    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528728    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528730    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528734    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528738    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528741    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528744    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528754    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528757    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528760    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528762    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528765    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528767    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528770    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528772    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528775    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.537627 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528777    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528780    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528782    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528785    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528787    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528791    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528794    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528797    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528799    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528801    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528804    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528806    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528809    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528811    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528813    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528816    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.538108 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.528838    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528872    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528876    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528879    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528881    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528883    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528886    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528889    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528891    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528894    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528896    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528899    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528901    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528904    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528906    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528908    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528911    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528913    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528915    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528918    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:09.538505 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528920    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528923    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528926    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528928    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528931    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528933    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528935    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528938    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528940    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528943    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528945    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528948    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528951    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528953    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528956    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528958    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528961    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528963    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528966    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528968    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:09.538962 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528971    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528973    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528976    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528978    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528980    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528983    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528986    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528989    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528991    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528993    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528996    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.528999    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529001    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529004    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529006    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529010    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529012    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529015    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529017    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:09.539446 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529019    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529022    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529024    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529027    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529030    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529034    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529036    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529039    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529041    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529043    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529046    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529048    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529051    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529053    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529055    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529058    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529060    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529063    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529065    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:09.539941 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529068    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529070    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529072    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529075    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529079    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529082    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529084    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529087    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529089    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529093    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529097    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529100    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529103    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529105    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529108    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529110    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529114    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529116    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529118    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529122    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:09.540402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529124    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529127    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529129    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529132    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529134    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529137    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529151    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529153    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:09.529156    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.529205    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.531162    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.532184    2572 server.go:1014] "Starting client certificate rotation"
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.532285    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.532316    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.534299    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:37:09.540886 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.536438    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:37:09.549823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.549805    2572 log.go:25] "Validated CRI v1 runtime API"
Nov 03 18:37:09.552812 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.552799    2572 log.go:25] "Validated CRI v1 image API"
Nov 03 18:37:09.553659 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.553630    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 18:37:09.558671 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.558652    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a00575e-a0e5-421b-a045-6c7c66e9f3dd:/dev/nvme0n1p3 a9b9e9a1-f123-47f2-85d3-ebb4d20e42bd:/dev/nvme0n1p4]
Nov 03 18:37:09.558741 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.558670    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 18:37:09.563574 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.563466    2572 manager.go:217] Machine: {Timestamp:2026-11-03 18:37:09.562190675 +0000 UTC m=+0.334178453 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3161297 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f858f5bf8eeefa83a34fbe47d76f4 SystemUUID:ec2f858f-5bf8-eeef-a83a-34fbe47d76f4 BootID:3d217963-b0ef-416b-8f44-2dec473f3f87 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d4:da:f5:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d4:da:f5:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:09:19:01:0f:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 18:37:09.563574 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.563573    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 18:37:09.563672 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.563644    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 18:37:09.566160 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.566126    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 18:37:09.566317 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.566162    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-45.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 18:37:09.566357 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.566320    2572 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 18:37:09.566357 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.566328    2572 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 18:37:09.566411 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.566364    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:37:09.566411 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.566383    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:37:09.567188 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.567178    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 18:37:09.569842 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.569831    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 18:37:09.569959 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.569950    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 18:37:09.570566 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.570555    2572 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 18:37:09.570595 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.570572    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 18:37:09.570595 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.570582    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 18:37:09.570595 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.570591    2572 kubelet.go:405] "Adding apiserver pod source"
Nov 03 18:37:09.570673 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.570599    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 18:37:09.570894 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.570882    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:37:09.573856 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.573834    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 18:37:09.578599 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.578587    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:37:09.579367 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579355    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 18:37:09.579415 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579385    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 18:37:09.579536 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579526    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 18:37:09.579569 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579542    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 18:37:09.579569 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579548    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 18:37:09.579569 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579554    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 18:37:09.579569 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579559    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 18:37:09.579569 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.579565    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 18:37:09.580580 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.580571    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 18:37:09.580580 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.580580    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 18:37:09.580634 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.580588    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 18:37:09.580634 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.580594    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 18:37:09.580634 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.580608    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 18:37:09.580634 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.580616    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 18:37:09.581323 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.581315    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 18:37:09.581359 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.581325    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 18:37:09.584973 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.584960    2572 server.go:1263] "Started kubelet"
Nov 03 18:37:09.585180 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.585017    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 18:37:09.585604 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.585072    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 18:37:09.585658 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.585629    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 18:37:09.585779 ip-10-0-143-45 systemd[1]: Started Kubernetes Kubelet.
Nov 03 18:37:09.586781 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.586705    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 18:37:09.587134 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.587122    2572 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 18:37:09.591577 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.591549    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 18:37:09.593116 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.593099    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 18:37:09.593632 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.593618    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 18:37:09.593751 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.593725    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:09.594512 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.594488    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 18:37:09.594512 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.594489    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 18:37:09.594641 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.594520    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 18:37:09.594641 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.594623    2572 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 18:37:09.594641 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.594631    2572 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 18:37:09.594758 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.594688    2572 factory.go:55] Registering systemd factory
Nov 03 18:37:09.594758 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.594725    2572 factory.go:223] Registration of the systemd container factory successfully
Nov 03 18:37:09.596660 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.596640    2572 factory.go:153] Registering CRI-O factory
Nov 03 18:37:09.596660 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.596661    2572 factory.go:223] Registration of the crio container factory successfully
Nov 03 18:37:09.596801 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.596714    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 18:37:09.596801 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.596730    2572 factory.go:103] Registering Raw factory
Nov 03 18:37:09.596801 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.596741    2572 manager.go:1199] Started watching for new ooms in manager
Nov 03 18:37:09.598076 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.598059    2572 manager.go:319] Starting recovery of all containers
Nov 03 18:37:09.606399 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.606385    2572 manager.go:324] Recovery completed
Nov 03 18:37:09.606702 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.606662    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 18:37:09.607961 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.607941    2572 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 03 18:37:09.610679 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.610665    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:09.612411 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.612381    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:09.612484 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.612426    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:09.612484 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.612443    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:09.612910 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.612900    2572 cpu_manager.go:226] "Starting" policy="none"
Nov 03 18:37:09.612910 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.612908    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 18:37:09.613001 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.612917    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:09.613001 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.612924    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 18:37:09.613614 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.613598    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:09.613667 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.613623    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:09.613667 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.613633    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:09.614061 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.614047    2572 policy_none.go:50] "Start"
Nov 03 18:37:09.614111 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.614063    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 18:37:09.614111 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.614078    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 18:37:09.615102 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.615092    2572 policy_none.go:44] "Start"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.660440    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.665268    2572 manager.go:341] "Starting Device Plugin manager"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.665311    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.665322    2572 server.go:87] "Starting device plugin registration server"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.665566    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.665581    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.665710    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.665865    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.665875    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.666467    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.666520    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.672261    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.672283    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.672300    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 18:37:09.673969 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.672328    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 18:37:09.734831 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.734810    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-45.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 18:37:09.734972 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.734844    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-45.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 18:37:09.736320 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.734835    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-45.us-west-2.compute.internal.18e49cde290177b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-45.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-45.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:09.584934833 +0000 UTC m=+0.356922610,LastTimestamp:2026-11-03 18:37:09.584934833 +0000 UTC m=+0.356922610,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-45.us-west-2.compute.internal,}"
Nov 03 18:37:09.765755 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.765697    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:09.766554 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.766538    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:09.766630 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.766569    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:09.766630 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.766580    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:09.766630 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.766604    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.773419 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.773383    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"]
Nov 03 18:37:09.773504 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.773445    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:09.774151 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.774114    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:09.774241 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.774165    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:09.774241 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.774177    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:09.775376 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.775364    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:09.775542 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.775524    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.775603 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.775563    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:09.776069 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.776051    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:09.776160 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.776059    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:09.776160 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.776087    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:09.776160 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.776100    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:09.776160 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.776101    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:09.776160 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.776112    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:09.777205 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.777192    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.777255 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.777213    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:09.777841 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.777824    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:09.777912 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.777858    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:09.777912 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.777873    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:09.796423 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.796400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.796507 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.796429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.796507 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.796446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e273692ac2c0053e598770db34f2c6e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"6e273692ac2c0053e598770db34f2c6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.804880 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.804861    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.808071 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.808052    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.823892 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.823868    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 18:37:09.823992 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.823934    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hqp9j"
Nov 03 18:37:09.828044 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.828016    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 18:37:09.828124 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.828059    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 18:37:09.828463 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.828446    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-45.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 18:37:09.828463 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.828452    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 03 18:37:09.873787 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.873706    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-45.us-west-2.compute.internal.18e49cde2aa4a930  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-45.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-45.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-45.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:09.612407088 +0000 UTC m=+0.384394869,LastTimestamp:2026-11-03 18:37:09.612407088 +0000 UTC m=+0.384394869,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-45.us-west-2.compute.internal,}"
Nov 03 18:37:09.886065 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:09.886043    2572 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-45.us-west-2.compute.internal\" not found" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.897453 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.897436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.897516 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.897470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.897516 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.897502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e273692ac2c0053e598770db34f2c6e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"6e273692ac2c0053e598770db34f2c6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.897579 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.897546    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.897579 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.897555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:09.897650 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:09.897558    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e273692ac2c0053e598770db34f2c6e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"6e273692ac2c0053e598770db34f2c6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:10.032770 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.032709    2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-45.us-west-2.compute.internal\" not found" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:10.087124 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.087104    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:10.088108 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.088094    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:10.088198 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.088122    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:10.088198 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.088132    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:10.088198 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.088171    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:10.107511 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.107494    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:10.112133 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.112117    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:10.142419 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.142402    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:10.142419 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.142419    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-45.us-west-2.compute.internal\": node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.253446 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.253418    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.353684 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.353614    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.454593 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.454560    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.531715 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.531682    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 18:37:10.532183 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.531838    2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://acc3b4ca8eba54ce98364924c8229c13-670cf6387112f978.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-45.us-west-2.compute.internal.18e49cde2aa5130e\": read tcp 10.0.143.45:38588->77.112.240.26:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-45.us-west-2.compute.internal.18e49cde2aa5130e  default   4961 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-45.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-45.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-45.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:09 +0000 UTC,LastTimestamp:2026-11-03 18:37:09.774170438 +0000 UTC m=+0.546158214,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-45.us-west-2.compute.internal,}"
Nov 03 18:37:10.554995 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.554974    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.655684 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.655620    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.756098 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.756062    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.827503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.827450    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:32:09 +0000 UTC" deadline="2028-08-16 10:56:17.868532142 +0000 UTC"
Nov 03 18:37:10.827503 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.827495    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15640h19m7.041041579s"
Nov 03 18:37:10.857201 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:10.857179    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:10.893086 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.893061    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:10.893967 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.893949    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:10.894032 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.893988    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:10.894032 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.893999    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:10.915837 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.915779    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:10.938936 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.938914    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:10.994560 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:10.994527    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:11.056119 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.056096    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:37:11.057032 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.057018    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:11.110292 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:11.110256    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e273692ac2c0053e598770db34f2c6e.slice/crio-65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7 WatchSource:0}: Error finding container 65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7: Status 404 returned error can't find the container with id 65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7
Nov 03 18:37:11.110586 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:11.110569    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5260ce34b15d34a9c6ca404e33bcf9fb.slice/crio-081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d WatchSource:0}: Error finding container 081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d: Status 404 returned error can't find the container with id 081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d
Nov 03 18:37:11.114790 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.114773    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:37:11.115557 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.115542    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:37:11.344764 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.344680    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:11.369967 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.369943    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:11.572246 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.572220    2572 apiserver.go:52] "Watching apiserver"
Nov 03 18:37:11.595133 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.595072    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 18:37:11.676925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.676867    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"6e273692ac2c0053e598770db34f2c6e","Type":"ContainerStarted","Data":"65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7"}
Nov 03 18:37:11.677924 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.677895    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"5260ce34b15d34a9c6ca404e33bcf9fb","Type":"ContainerStarted","Data":"081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d"}
Nov 03 18:37:11.684987 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.684961    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 18:37:11.685321 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.685302    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-rxtfb","openshift-multus/multus-additional-cni-plugins-lx772","openshift-multus/network-metrics-daemon-7sqm4","openshift-network-diagnostics/network-check-target-m76pn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx","openshift-network-operator/iptables-alerter-x54sg","kube-system/konnectivity-agent-b2gtz","kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal","openshift-image-registry/node-ca-z7fg9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal","openshift-multus/multus-psxdh","kube-system/global-pull-secret-syncer-xr6hd","openshift-cluster-node-tuning-operator/tuned-chzws","openshift-dns/node-resolver-vncch"]
Nov 03 18:37:11.687748 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.687729    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:11.690823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.690793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.691850 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.691831    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:11.691959 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.691934    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:11.693026 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.693007    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:11.693108 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.693088    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:11.694146 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.694115    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.695313 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.695290    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.697608 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.697590    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.699269 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.699250    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.700569 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.700552    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.701876 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.701855    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.703152 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.703130    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.704404 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.704386    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.704497 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.704453    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:11.707232 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707195    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-run\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.707330 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707233    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9fsn\" (UniqueName: \"kubernetes.io/projected/1870fed7-afe2-43c7-ad6d-be96cf7624f2-kube-api-access-n9fsn\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:11.707330 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707294    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:11.707451 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707331    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43940503-68c4-41e5-9656-4e1fe7837110-agent-certs\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:11.707451 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-modprobe-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.707451 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707388    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.707451 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707424    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-tuned\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.707661 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-tmp\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.707661 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707476    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-socket-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.707661 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707514    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21ec2464-111d-4061-9938-44d42bea81b7-iptables-alerter-script\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.707661 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707558    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21ec2464-111d-4061-9938-44d42bea81b7-host-slash\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.707661 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707606    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43940503-68c4-41e5-9656-4e1fe7837110-konnectivity-ca\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:11.707661 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-kubernetes\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-conf\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-lib-modules\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707707    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-host\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsnmx\" (UniqueName: \"kubernetes.io/projected/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-kube-api-access-dsnmx\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707811    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-device-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-systemd\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707917    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-sys\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-var-lib-kubelet\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.707975    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.708003    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-sys-fs\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.708038    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhgv\" (UniqueName: \"kubernetes.io/projected/21ec2464-111d-4061-9938-44d42bea81b7-kube-api-access-4bhgv\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.708081 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.708064    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:11.708783 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.708090    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-registration-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.708783 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.708129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwzg\" (UniqueName: \"kubernetes.io/projected/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kube-api-access-vjwzg\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.708783 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.708164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysconfig\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.745919 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.745899    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 18:37:11.745919 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.745915    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 18:37:11.746040 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.745966    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczj\""
Nov 03 18:37:11.747725 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.747707    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 18:37:11.747833 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.747821    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:37:11.747904 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.747892    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d4mhq\""
Nov 03 18:37:11.750836 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.750823    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 18:37:11.751064 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.751052    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 18:37:11.751207 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.751192    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 18:37:11.751638 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.751626    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cxqpk\""
Nov 03 18:37:11.752138 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.752124    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zsb8x\""
Nov 03 18:37:11.752230 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.752203    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 18:37:11.752290 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.752275    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:37:11.752403 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.752392    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 18:37:11.754594 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.754580    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 18:37:11.754631 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.754600    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 18:37:11.754631 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.754607    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 18:37:11.754631 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.754627    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fft52\""
Nov 03 18:37:11.756060 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.756046    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 18:37:11.756268 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.756253    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 18:37:11.756268 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.756264    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxjph\""
Nov 03 18:37:11.756333 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.756281    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 18:37:11.756368 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.756349    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 18:37:11.756471 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.756459    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 18:37:11.758697 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.756770    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 18:37:11.795945 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.795915    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 18:37:11.808601 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.808660 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808608    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.808660 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808627    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7wh\" (UniqueName: \"kubernetes.io/projected/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-kube-api-access-tv7wh\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.808660 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.808794 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808663    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-env-overrides\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.808794 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-multus\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.808794 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-kubernetes\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.808964 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-conf\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.808964 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-host\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.808964 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808891    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-kubernetes\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.808964 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-dbus\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.808964 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808929    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c167fdc3-2279-434d-bb1a-022b46d24b2c-host\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.808964 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808958    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-host\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.809141 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808969    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-conf\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.809141 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.808997    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.809141 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809036    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-netns\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.809141 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809057    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-device-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.809141 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809103    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-device-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.809141 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809107    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-sys\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.809141 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809135    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/038e7f0f-0691-4006-aaee-e8b99231be0a-hosts-file\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809189    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-sys\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-kubelet-config\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809235    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-netns\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809255    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809272    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-sys-fs\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809288    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038e7f0f-0691-4006-aaee-e8b99231be0a-tmp-dir\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809303    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-systemd-units\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809319    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-script-lib\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-socket-dir-parent\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809333    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-sys-fs\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809303    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.809372 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809357    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-multus-certs\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsrc\" (UniqueName: \"kubernetes.io/projected/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-kube-api-access-jrsrc\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809403    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809422    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-conf-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cnibin\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809475    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwzg\" (UniqueName: \"kubernetes.io/projected/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kube-api-access-vjwzg\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pwsl\" (UniqueName: \"kubernetes.io/projected/c167fdc3-2279-434d-bb1a-022b46d24b2c-kube-api-access-2pwsl\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809617    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809643    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-system-cni-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysconfig\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.809729 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809731    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysconfig\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9fsn\" (UniqueName: \"kubernetes.io/projected/1870fed7-afe2-43c7-ad6d-be96cf7624f2-kube-api-access-n9fsn\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43940503-68c4-41e5-9656-4e1fe7837110-agent-certs\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809787    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-modprobe-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809813    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-tuned\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809826    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-tmp\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-socket-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809856    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-systemd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809875    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-etc-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-bin\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-multus-daemon-config\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809971    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-modprobe-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809976    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.809989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21ec2464-111d-4061-9938-44d42bea81b7-iptables-alerter-script\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810024    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21ec2464-111d-4061-9938-44d42bea81b7-host-slash\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810042    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-socket-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.810147 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43940503-68c4-41e5-9656-4e1fe7837110-konnectivity-ca\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810096    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-lib-modules\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810100    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21ec2464-111d-4061-9938-44d42bea81b7-host-slash\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnmx\" (UniqueName: \"kubernetes.io/projected/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-kube-api-access-dsnmx\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810208    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810216    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-var-lib-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810236    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-lib-modules\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-log-socket\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810275    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-hostroot\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810388    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-systemd\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-var-lib-kubelet\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c167fdc3-2279-434d-bb1a-022b46d24b2c-serviceca\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810522    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-slash\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810546    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-ovn\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810545    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-systemd\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810601    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-kubelet\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.810925 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810610    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-var-lib-kubelet\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810634    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhgv\" (UniqueName: \"kubernetes.io/projected/21ec2464-111d-4061-9938-44d42bea81b7-kube-api-access-4bhgv\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810716    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-bin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21ec2464-111d-4061-9938-44d42bea81b7-iptables-alerter-script\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810718    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810717    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43940503-68c4-41e5-9656-4e1fe7837110-konnectivity-ca\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-registration-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810826    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810856    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-registration-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810857    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-kubelet\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810909    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-node-log\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-netd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.810981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-config\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.811022    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.811103    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:12.311081926 +0000 UTC m=+3.083069715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:11.811927 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-cnibin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811163    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6b4\" (UniqueName: \"kubernetes.io/projected/038e7f0f-0691-4006-aaee-e8b99231be0a-kube-api-access-dr6b4\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811193    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-etc-kubernetes\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-run\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811295    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811374    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-run\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811374    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovn-node-metrics-cert\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811414    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-system-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811474    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-cni-binary-copy\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lltc4\" (UniqueName: \"kubernetes.io/projected/878a9a2c-5584-4c23-8b39-754abed5d341-kube-api-access-lltc4\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.812763 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.811542    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-k8s-cni-cncf-io\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.813443 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.813421    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-tmp\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.813485 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.813454    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-tuned\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.813645 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.813631    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43940503-68c4-41e5-9656-4e1fe7837110-agent-certs\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:11.850633 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.850575    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 18:37:11.850725 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.850636    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mqbrj"
Nov 03 18:37:11.854629 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854611    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 18:37:11.854720 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854704    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 18:37:11.854787 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854750    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 18:37:11.854787 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854768    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 18:37:11.854871 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854804    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 18:37:11.854871 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854835    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lzn26\""
Nov 03 18:37:11.854943 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854874    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 18:37:11.855014 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.854999    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r88kn\""
Nov 03 18:37:11.855048 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.855013    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 18:37:11.855048 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.855016    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 18:37:11.855048 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.855035    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-btttq\""
Nov 03 18:37:11.874719 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.874665    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:37:11.874719 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.874689    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:37:11.874719 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.874702    2572 projected.go:196] Error preparing data for projected volume kube-api-access-b5lhj for pod openshift-network-diagnostics/network-check-target-m76pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:11.874938 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.874774    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj podName:865029c1-7176-4b5c-aa12-f4a237af8e60 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:12.37475476 +0000 UTC m=+3.146742546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b5lhj" (UniqueName: "kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj") pod "network-check-target-m76pn" (UID: "865029c1-7176-4b5c-aa12-f4a237af8e60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:11.876461 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.876441    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnmx\" (UniqueName: \"kubernetes.io/projected/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-kube-api-access-dsnmx\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:11.876558 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.876470    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhgv\" (UniqueName: \"kubernetes.io/projected/21ec2464-111d-4061-9938-44d42bea81b7-kube-api-access-4bhgv\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:11.876650 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.876631    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwzg\" (UniqueName: \"kubernetes.io/projected/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kube-api-access-vjwzg\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:11.876688 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.876660    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9fsn\" (UniqueName: \"kubernetes.io/projected/1870fed7-afe2-43c7-ad6d-be96cf7624f2-kube-api-access-n9fsn\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:11.912600 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912568    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/038e7f0f-0691-4006-aaee-e8b99231be0a-hosts-file\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.912676 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912604    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-kubelet-config\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.912676 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-netns\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.912676 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038e7f0f-0691-4006-aaee-e8b99231be0a-tmp-dir\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.912676 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912668    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-systemd-units\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912700    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/038e7f0f-0691-4006-aaee-e8b99231be0a-hosts-file\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912692    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-script-lib\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912718    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-netns\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912720    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-kubelet-config\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912753    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-systemd-units\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-socket-dir-parent\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912793    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-socket-dir-parent\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.912823 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912793    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-multus-certs\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912825    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-multus-certs\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsrc\" (UniqueName: \"kubernetes.io/projected/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-kube-api-access-jrsrc\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912860    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912885    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-conf-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912909    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cnibin\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912935    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pwsl\" (UniqueName: \"kubernetes.io/projected/c167fdc3-2279-434d-bb1a-022b46d24b2c-kube-api-access-2pwsl\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912984    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-system-cni-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.912994    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038e7f0f-0691-4006-aaee-e8b99231be0a-tmp-dir\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913009    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913010    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913052    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913055    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-conf-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913075    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-systemd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-etc-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-bin\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.913463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913134    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-multus-daemon-config\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913012    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cnibin\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913061    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-system-cni-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913112    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-systemd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913178    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-etc-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-var-lib-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913192    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-bin\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-log-socket\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-hostroot\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913288    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-script-lib\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c167fdc3-2279-434d-bb1a-022b46d24b2c-serviceca\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913310    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-var-lib-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913330    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-slash\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913343    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-hostroot\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913311    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-log-socket\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-ovn\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913389    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-ovn\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913392    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-kubelet\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.914386 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913437    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-kubelet\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913440    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-bin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913469    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-bin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913400    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-slash\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913526    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-kubelet\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913549    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-node-log\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913599    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-node-log\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.913616    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913642    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-kubelet\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913677    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-netd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:11.913686    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret podName:793f3faa-8090-4008-a15c-13202c13e875 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:12.413673029 +0000 UTC m=+3.185660796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret") pod "global-pull-secret-syncer-xr6hd" (UID: "793f3faa-8090-4008-a15c-13202c13e875") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-config\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913734    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-cnibin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6b4\" (UniqueName: \"kubernetes.io/projected/038e7f0f-0691-4006-aaee-e8b99231be0a-kube-api-access-dr6b4\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913770    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-etc-kubernetes\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913712    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-netd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915239 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913817    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-cnibin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913840    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-etc-kubernetes\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913870    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovn-node-metrics-cert\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913885    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-system-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913907    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-cni-binary-copy\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913952    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-system-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.913987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lltc4\" (UniqueName: \"kubernetes.io/projected/878a9a2c-5584-4c23-8b39-754abed5d341-kube-api-access-lltc4\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-k8s-cni-cncf-io\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7wh\" (UniqueName: \"kubernetes.io/projected/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-kube-api-access-tv7wh\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914128    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914165    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914155    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-env-overrides\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-multus\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914234    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-dbus\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.915742 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914100    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-k8s-cni-cncf-io\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914258    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c167fdc3-2279-434d-bb1a-022b46d24b2c-host\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914285    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-netns\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914382    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c167fdc3-2279-434d-bb1a-022b46d24b2c-serviceca\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914390    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-netns\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914389    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-multus-daemon-config\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914411    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-dbus\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914430    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-multus\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914432    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c167fdc3-2279-434d-bb1a-022b46d24b2c-host\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914450    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914480    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914608    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914764    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914808    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.914939    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-env-overrides\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.915007    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-config\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.916208 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.915024    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-cni-binary-copy\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.916711 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.916692    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovn-node-metrics-cert\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.979621 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.979592    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsrc\" (UniqueName: \"kubernetes.io/projected/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-kube-api-access-jrsrc\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:11.979806 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.979723    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pwsl\" (UniqueName: \"kubernetes.io/projected/c167fdc3-2279-434d-bb1a-022b46d24b2c-kube-api-access-2pwsl\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:11.979895 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.979871    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6b4\" (UniqueName: \"kubernetes.io/projected/038e7f0f-0691-4006-aaee-e8b99231be0a-kube-api-access-dr6b4\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:11.980975 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.980950    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7wh\" (UniqueName: \"kubernetes.io/projected/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-kube-api-access-tv7wh\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:11.981067 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.981028    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lltc4\" (UniqueName: \"kubernetes.io/projected/878a9a2c-5584-4c23-8b39-754abed5d341-kube-api-access-lltc4\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:11.999236 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:11.999201    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:12.006333 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.006314    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:12.011310 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.011288    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43940503_68c4_41e5_9656_4e1fe7837110.slice/crio-e5b903e852e4a4671b4df9924549604a77a090a568fd3ab228e9fb6c420cb3be WatchSource:0}: Error finding container e5b903e852e4a4671b4df9924549604a77a090a568fd3ab228e9fb6c420cb3be: Status 404 returned error can't find the container with id e5b903e852e4a4671b4df9924549604a77a090a568fd3ab228e9fb6c420cb3be
Nov 03 18:37:12.015363 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.015341    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:12.016295 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.016266    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf6e514_f74c_4e4d_b8e9_1180fc6a7ec8.slice/crio-c4b37c5d598179d7d7ed015f0fc5f4795ea903894f8e5b443128282878dc90f0 WatchSource:0}: Error finding container c4b37c5d598179d7d7ed015f0fc5f4795ea903894f8e5b443128282878dc90f0: Status 404 returned error can't find the container with id c4b37c5d598179d7d7ed015f0fc5f4795ea903894f8e5b443128282878dc90f0
Nov 03 18:37:12.018026 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.018010    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:12.024027 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.024008    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:12.025032 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.025011    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344ba3f2_5bd5_4e78_bd7d_101ad41f998e.slice/crio-16cd488feffbe269e60eda61387381df8760e22e4cdb49dee7ed3cd507393082 WatchSource:0}: Error finding container 16cd488feffbe269e60eda61387381df8760e22e4cdb49dee7ed3cd507393082: Status 404 returned error can't find the container with id 16cd488feffbe269e60eda61387381df8760e22e4cdb49dee7ed3cd507393082
Nov 03 18:37:12.025714 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.025698    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ec2464_111d_4061_9938_44d42bea81b7.slice/crio-f4b07bde72d02f008a095963395e92a5955bc92700c7df515365c015a643a57e WatchSource:0}: Error finding container f4b07bde72d02f008a095963395e92a5955bc92700c7df515365c015a643a57e: Status 404 returned error can't find the container with id f4b07bde72d02f008a095963395e92a5955bc92700c7df515365c015a643a57e
Nov 03 18:37:12.032752 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.032726    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc167fdc3_2279_434d_bb1a_022b46d24b2c.slice/crio-fb8ab53b2c67e8cefe666b75b67a16233bfdbe811c713222352180690b2cf3d6 WatchSource:0}: Error finding container fb8ab53b2c67e8cefe666b75b67a16233bfdbe811c713222352180690b2cf3d6: Status 404 returned error can't find the container with id fb8ab53b2c67e8cefe666b75b67a16233bfdbe811c713222352180690b2cf3d6
Nov 03 18:37:12.037192 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.037173    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:12.044186 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.044167    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:12.044300 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.044283    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8903afd0_dd9e_42e1_9aa0_d228f3ef611c.slice/crio-22b221e79df93b7ed2a9c787d47d152fce8fdd2438ac168e72ebe9eedaa29e92 WatchSource:0}: Error finding container 22b221e79df93b7ed2a9c787d47d152fce8fdd2438ac168e72ebe9eedaa29e92: Status 404 returned error can't find the container with id 22b221e79df93b7ed2a9c787d47d152fce8fdd2438ac168e72ebe9eedaa29e92
Nov 03 18:37:12.045410 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.045396    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:12.046870 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.046851    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psxdh"
Nov 03 18:37:12.051790 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.051701    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038e7f0f_0691_4006_aaee_e8b99231be0a.slice/crio-fcad5aec9f095876d6ab9d8524fefed8d0c03b7195a6ce1449c0cf9e3d725a71 WatchSource:0}: Error finding container fcad5aec9f095876d6ab9d8524fefed8d0c03b7195a6ce1449c0cf9e3d725a71: Status 404 returned error can't find the container with id fcad5aec9f095876d6ab9d8524fefed8d0c03b7195a6ce1449c0cf9e3d725a71
Nov 03 18:37:12.054665 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.054646    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3065ca9e_9bb8_4dd6_a39e_0c36efb9d866.slice/crio-ee53dcab1461cda6f09acdcf9dc0e732fb3f5f9340d29af4ab95cd5867f0bed0 WatchSource:0}: Error finding container ee53dcab1461cda6f09acdcf9dc0e732fb3f5f9340d29af4ab95cd5867f0bed0: Status 404 returned error can't find the container with id ee53dcab1461cda6f09acdcf9dc0e732fb3f5f9340d29af4ab95cd5867f0bed0
Nov 03 18:37:12.056890 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:12.056860    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878a9a2c_5584_4c23_8b39_754abed5d341.slice/crio-017f22f4a10e222aacb30b63536e86e280461b31fb3664381eba47513819142d WatchSource:0}: Error finding container 017f22f4a10e222aacb30b63536e86e280461b31fb3664381eba47513819142d: Status 404 returned error can't find the container with id 017f22f4a10e222aacb30b63536e86e280461b31fb3664381eba47513819142d
Nov 03 18:37:12.318605 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.318461    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:12.318766 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.318616    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:12.318766 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.318678    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:13.318661264 +0000 UTC m=+4.090649035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:12.419000 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.418966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:12.419000 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.419011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:12.419281 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.419151    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:37:12.419281 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.419149    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:12.419281 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.419171    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:37:12.419281 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.419210    2572 projected.go:196] Error preparing data for projected volume kube-api-access-b5lhj for pod openshift-network-diagnostics/network-check-target-m76pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:12.419281 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.419260    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret podName:793f3faa-8090-4008-a15c-13202c13e875 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:13.419239208 +0000 UTC m=+4.191226989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret") pod "global-pull-secret-syncer-xr6hd" (UID: "793f3faa-8090-4008-a15c-13202c13e875") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:12.419281 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:12.419283    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj podName:865029c1-7176-4b5c-aa12-f4a237af8e60 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:13.419272005 +0000 UTC m=+4.191259787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5lhj" (UniqueName: "kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj") pod "network-check-target-m76pn" (UID: "865029c1-7176-4b5c-aa12-f4a237af8e60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:12.681980 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.681891    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx" event={"ID":"344ba3f2-5bd5-4e78-bd7d-101ad41f998e","Type":"ContainerStarted","Data":"16cd488feffbe269e60eda61387381df8760e22e4cdb49dee7ed3cd507393082"}
Nov 03 18:37:12.683676 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.683646    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b2gtz" event={"ID":"43940503-68c4-41e5-9656-4e1fe7837110","Type":"ContainerStarted","Data":"e5b903e852e4a4671b4df9924549604a77a090a568fd3ab228e9fb6c420cb3be"}
Nov 03 18:37:12.685006 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.684966    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerStarted","Data":"ee53dcab1461cda6f09acdcf9dc0e732fb3f5f9340d29af4ab95cd5867f0bed0"}
Nov 03 18:37:12.686450 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.686418    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"22b221e79df93b7ed2a9c787d47d152fce8fdd2438ac168e72ebe9eedaa29e92"}
Nov 03 18:37:12.687896 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.687839    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-chzws" event={"ID":"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8","Type":"ContainerStarted","Data":"c4b37c5d598179d7d7ed015f0fc5f4795ea903894f8e5b443128282878dc90f0"}
Nov 03 18:37:12.689362 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.689330    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psxdh" event={"ID":"878a9a2c-5584-4c23-8b39-754abed5d341","Type":"ContainerStarted","Data":"017f22f4a10e222aacb30b63536e86e280461b31fb3664381eba47513819142d"}
Nov 03 18:37:12.690662 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.690619    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vncch" event={"ID":"038e7f0f-0691-4006-aaee-e8b99231be0a","Type":"ContainerStarted","Data":"fcad5aec9f095876d6ab9d8524fefed8d0c03b7195a6ce1449c0cf9e3d725a71"}
Nov 03 18:37:12.692004 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.691977    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7fg9" event={"ID":"c167fdc3-2279-434d-bb1a-022b46d24b2c","Type":"ContainerStarted","Data":"fb8ab53b2c67e8cefe666b75b67a16233bfdbe811c713222352180690b2cf3d6"}
Nov 03 18:37:12.693280 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.693258    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x54sg" event={"ID":"21ec2464-111d-4061-9938-44d42bea81b7","Type":"ContainerStarted","Data":"f4b07bde72d02f008a095963395e92a5955bc92700c7df515365c015a643a57e"}
Nov 03 18:37:12.853315 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.853228    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:11 +0000 UTC" deadline="2028-07-02 05:40:39.498967817 +0000 UTC"
Nov 03 18:37:12.853315 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:12.853309    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14555h3m26.645663494s"
Nov 03 18:37:13.328441 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.328349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:13.328616 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.328553    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:13.328689 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.328625    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:15.328604144 +0000 UTC m=+6.100591932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:13.429067 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.429017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:13.429258 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.429118    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:13.429405 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.429354    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:37:13.429405 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.429387    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:37:13.429405 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.429403    2572 projected.go:196] Error preparing data for projected volume kube-api-access-b5lhj for pod openshift-network-diagnostics/network-check-target-m76pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:13.429588 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.429489    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj podName:865029c1-7176-4b5c-aa12-f4a237af8e60 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:15.429468148 +0000 UTC m=+6.201455931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5lhj" (UniqueName: "kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj") pod "network-check-target-m76pn" (UID: "865029c1-7176-4b5c-aa12-f4a237af8e60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:13.429661 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.429603    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:13.429661 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.429646    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret podName:793f3faa-8090-4008-a15c-13202c13e875 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:15.429633534 +0000 UTC m=+6.201621315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret") pod "global-pull-secret-syncer-xr6hd" (UID: "793f3faa-8090-4008-a15c-13202c13e875") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:13.675663 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.675611    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:13.675817 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.675634    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:13.675817 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.675795    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:13.675944 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.675828    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:13.675998 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.675943    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:13.676059 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:13.676023    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:13.854463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.854368    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:11 +0000 UTC" deadline="2028-05-11 21:50:32.505753331 +0000 UTC"
Nov 03 18:37:13.854463 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:13.854405    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13323h13m18.651353728s"
Nov 03 18:37:14.711189 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:14.710910    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"6e273692ac2c0053e598770db34f2c6e","Type":"ContainerStarted","Data":"48796a8eb56aab8e628ffc7430dcdc40437cca7a43d4e7196c4cb84b6266d2a8"}
Nov 03 18:37:14.722238 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:14.722207    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"5260ce34b15d34a9c6ca404e33bcf9fb","Type":"ContainerStarted","Data":"c006ed3495eba84605737fdad66af604926e5b558c92406b9164a3b4d8ecc757"}
Nov 03 18:37:14.948367 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:14.945432    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal" podStartSLOduration=3.945413778 podStartE2EDuration="3.945413778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:14.826849044 +0000 UTC m=+5.598836832" watchObservedRunningTime="2026-11-03 18:37:14.945413778 +0000 UTC m=+5.717401569"
Nov 03 18:37:15.357643 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.357605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:15.357829 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.357814    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:15.357904 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.357881    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:19.357861654 +0000 UTC m=+10.129849433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:15.458680 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.458606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:15.458680 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.458668    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:15.458906 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.458865    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:37:15.458906 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.458887    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:37:15.458906 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.458899    2572 projected.go:196] Error preparing data for projected volume kube-api-access-b5lhj for pod openshift-network-diagnostics/network-check-target-m76pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:15.459010 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.458957    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj podName:865029c1-7176-4b5c-aa12-f4a237af8e60 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:19.458940348 +0000 UTC m=+10.230928129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5lhj" (UniqueName: "kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj") pod "network-check-target-m76pn" (UID: "865029c1-7176-4b5c-aa12-f4a237af8e60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:15.459050 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.459032    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:15.459081 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.459065    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret podName:793f3faa-8090-4008-a15c-13202c13e875 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:19.459054701 +0000 UTC m=+10.231042482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret") pod "global-pull-secret-syncer-xr6hd" (UID: "793f3faa-8090-4008-a15c-13202c13e875") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:15.675364 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.674855    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:15.675364 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.674979    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:15.675364 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.675047    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:15.675364 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.675113    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:15.675364 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.675188    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:15.675364 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:15.675321    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:15.729340 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.728427    2572 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="5dee7e1af06492cedbe91b3e8fb398d534a6f716b1aeaeb5f92f5d48ba15d2c6" exitCode=0
Nov 03 18:37:15.729340 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.728517    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerDied","Data":"5dee7e1af06492cedbe91b3e8fb398d534a6f716b1aeaeb5f92f5d48ba15d2c6"}
Nov 03 18:37:15.736367 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.736282    2572 generic.go:356] "Generic (PLEG): container finished" podID="5260ce34b15d34a9c6ca404e33bcf9fb" containerID="c006ed3495eba84605737fdad66af604926e5b558c92406b9164a3b4d8ecc757" exitCode=0
Nov 03 18:37:15.736468 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:15.736379    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"5260ce34b15d34a9c6ca404e33bcf9fb","Type":"ContainerDied","Data":"c006ed3495eba84605737fdad66af604926e5b558c92406b9164a3b4d8ecc757"}
Nov 03 18:37:16.744349 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:16.744292    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"5260ce34b15d34a9c6ca404e33bcf9fb","Type":"ContainerStarted","Data":"8f7abd582f3ee15210d7a75c24321976f6b5a4c735e87611bcede3fa827c707d"}
Nov 03 18:37:16.861694 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:16.861637    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" podStartSLOduration=5.861619229 podStartE2EDuration="5.861619229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:16.860505001 +0000 UTC m=+7.632492815" watchObservedRunningTime="2026-11-03 18:37:16.861619229 +0000 UTC m=+7.633607016"
Nov 03 18:37:17.675665 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:17.675631    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:17.675827 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:17.675766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:17.676084 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:17.676069    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:17.676179 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:17.676132    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:17.676235 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:17.676205    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:17.676301 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:17.676276    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:17.753693 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:17.753609    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b2gtz" event={"ID":"43940503-68c4-41e5-9656-4e1fe7837110","Type":"ContainerStarted","Data":"a040812849df8500b5ca457c7b5264a076e150d4ff454418f132f3aa302e69db"}
Nov 03 18:37:17.874522 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:17.874351    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b2gtz" podStartSLOduration=3.058367396 podStartE2EDuration="7.87432981s" totalImagesPullingTime="4.815962414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:17.873340058 +0000 UTC m=+8.645327845" watchObservedRunningTime="2026-11-03 18:37:17.87432981 +0000 UTC m=+8.646317602"
Nov 03 18:37:18.244351 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:18.244269    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:18.245301 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:18.245096    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:18.758978 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:18.758896    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z7fg9" event={"ID":"c167fdc3-2279-434d-bb1a-022b46d24b2c","Type":"ContainerStarted","Data":"a0a2edf31858a0fc70509a8dfec7bf639b8c3c986559ff501dead238291b09c3"}
Nov 03 18:37:18.763166 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:18.763124    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx" event={"ID":"344ba3f2-5bd5-4e78-bd7d-101ad41f998e","Type":"ContainerStarted","Data":"ab575ab9f98dd400a0cb6f1a1c56ad91f8c5ff8b31b2d4d62f14c2abbac3513d"}
Nov 03 18:37:19.393732 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:19.393678    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:19.393914 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.393882    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:19.393993 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.393975    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:27.393956746 +0000 UTC m=+18.165944524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:19.494678    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:19.494722    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.494846    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.494864    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.494877    2572 projected.go:196] Error preparing data for projected volume kube-api-access-b5lhj for pod openshift-network-diagnostics/network-check-target-m76pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.494924    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj podName:865029c1-7176-4b5c-aa12-f4a237af8e60 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:27.494911759 +0000 UTC m=+18.266899523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5lhj" (UniqueName: "kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj") pod "network-check-target-m76pn" (UID: "865029c1-7176-4b5c-aa12-f4a237af8e60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.495301    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:19.495382 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.495342    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret podName:793f3faa-8090-4008-a15c-13202c13e875 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:27.495331446 +0000 UTC m=+18.267319361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret") pod "global-pull-secret-syncer-xr6hd" (UID: "793f3faa-8090-4008-a15c-13202c13e875") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:19.676020 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:19.675365    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:19.676020 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.675490    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:19.676020 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:19.675826    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:19.676020 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.675909    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:19.676548 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:19.676088    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:19.676548 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:19.676219    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:19.766496 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:19.766223    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:37:20.781755 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:20.781714    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psxdh" event={"ID":"878a9a2c-5584-4c23-8b39-754abed5d341","Type":"ContainerStarted","Data":"95c0d847756cc06ead32c91497b9962d3a61cdaf280d268ef916d3146f9011ba"}
Nov 03 18:37:20.786968 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:20.786932    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 18:37:20.901546 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:20.901304    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z7fg9" podStartSLOduration=4.722913385 podStartE2EDuration="10.901282634s" totalImagesPullingTime="6.178369249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:18.876693602 +0000 UTC m=+9.648681388" watchObservedRunningTime="2026-11-03 18:37:20.901282634 +0000 UTC m=+11.673270418"
Nov 03 18:37:21.673741 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.673503    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:21.673861 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.673513    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:21.673916 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:21.673851    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:21.673916 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:21.673906    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:21.673981 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.673573    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:21.673981 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:21.673977    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:21.679847 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.679738    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:37:20.786952509Z","UUID":"f149b3e3-e730-4da1-946a-02259399a6ef","Handler":null,"Name":"","Endpoint":""}
Nov 03 18:37:21.681396 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.681356    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 18:37:21.681396 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.681385    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 18:37:21.786038 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.786004    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx" event={"ID":"344ba3f2-5bd5-4e78-bd7d-101ad41f998e","Type":"ContainerStarted","Data":"f881166970bc1d8a81df58a7228db39e55aa5a9456a7a279dd6f0c1e542b84ac"}
Nov 03 18:37:21.788518 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.788485    2572 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="144eaebf0cfcf6641080fcc8c56a43f8f72b8e74a500a52f0939df1d38f0f585" exitCode=0
Nov 03 18:37:21.788649 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.788585    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerDied","Data":"144eaebf0cfcf6641080fcc8c56a43f8f72b8e74a500a52f0939df1d38f0f585"}
Nov 03 18:37:21.790744 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.790714    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vncch" event={"ID":"038e7f0f-0691-4006-aaee-e8b99231be0a","Type":"ContainerStarted","Data":"bbddd0b15f87447546048d468ec70a006a0b6e88b21875a57f1984238cade4fa"}
Nov 03 18:37:21.920057 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:21.920003    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-psxdh" podStartSLOduration=3.324339416 podStartE2EDuration="11.919990863s" totalImagesPullingTime="8.595651447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:20.902311481 +0000 UTC m=+11.674299268" watchObservedRunningTime="2026-11-03 18:37:21.919990863 +0000 UTC m=+12.691978652"
Nov 03 18:37:22.035524 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:22.035469    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vncch" podStartSLOduration=3.430353548 podStartE2EDuration="12.035456166s" totalImagesPullingTime="8.605102618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:22.035299325 +0000 UTC m=+12.807287113" watchObservedRunningTime="2026-11-03 18:37:22.035456166 +0000 UTC m=+12.807443951"
Nov 03 18:37:22.795174 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:22.795118    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x54sg" event={"ID":"21ec2464-111d-4061-9938-44d42bea81b7","Type":"ContainerStarted","Data":"2208e4941184426a3243e084c8d3e7d13b1a4ac36bd1df3ad682cef00bbb2d3f"}
Nov 03 18:37:22.910822 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:22.910760    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x54sg" podStartSLOduration=4.242771467 podStartE2EDuration="12.910746106s" totalImagesPullingTime="8.667974639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:22.910706273 +0000 UTC m=+13.682694060" watchObservedRunningTime="2026-11-03 18:37:22.910746106 +0000 UTC m=+13.682733892"
Nov 03 18:37:23.672796 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.672703    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:23.672796 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.672767    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:23.672992 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.672775    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:23.672992 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:23.672915    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:23.673098 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:23.673020    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:23.673237 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:23.673097    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:23.799324 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.799286    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-chzws" event={"ID":"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8","Type":"ContainerStarted","Data":"799bc336b4b6cff7152d60038d079432dcf8fa1ba7388ebe4adde7a702a71048"}
Nov 03 18:37:23.801720 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.801693    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx" event={"ID":"344ba3f2-5bd5-4e78-bd7d-101ad41f998e","Type":"ContainerStarted","Data":"34f85066a708666f419cf9820419da2958a916a0038da00cf42ecd050e1ad9ba"}
Nov 03 18:37:23.804063 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.804030    2572 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="9a6c63cf165c7db8691ffaceb5eca53f0d12f3836fea5538ef609217d07a5951" exitCode=0
Nov 03 18:37:23.804186 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.804115    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerDied","Data":"9a6c63cf165c7db8691ffaceb5eca53f0d12f3836fea5538ef609217d07a5951"}
Nov 03 18:37:23.916935 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:23.916873    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-chzws" podStartSLOduration=2.987884098 podStartE2EDuration="13.91685575s" totalImagesPullingTime="10.928971652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:23.91676835 +0000 UTC m=+14.688756137" watchObservedRunningTime="2026-11-03 18:37:23.91685575 +0000 UTC m=+14.688843538"
Nov 03 18:37:24.161840 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:24.161744    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx" podStartSLOduration=3.26895757 podStartE2EDuration="14.161728833s" totalImagesPullingTime="10.892771263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:24.161227703 +0000 UTC m=+14.933215502" watchObservedRunningTime="2026-11-03 18:37:24.161728833 +0000 UTC m=+14.933716619"
Nov 03 18:37:25.673124 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:25.673046    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:25.673124 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:25.673082    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:25.673124 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:25.673081    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:25.673704 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:25.673218    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:25.673704 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:25.673543    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:25.673704 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:25.673594    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:25.811444 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:25.811256    2572 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="3734225d2fb7a07101f2e9b622fee8325099f9ffe063b6b963549720fe792544" exitCode=0
Nov 03 18:37:25.811444 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:25.811339    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerDied","Data":"3734225d2fb7a07101f2e9b622fee8325099f9ffe063b6b963549720fe792544"}
Nov 03 18:37:27.453966 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.453936    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:27.454349 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.454134    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:27.454349 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.454199    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:43.454185576 +0000 UTC m=+34.226173340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.554506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.554599    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.554727    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.554788    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret podName:793f3faa-8090-4008-a15c-13202c13e875 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:43.554771697 +0000 UTC m=+34.326759476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret") pod "global-pull-secret-syncer-xr6hd" (UID: "793f3faa-8090-4008-a15c-13202c13e875") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.554881    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.554901    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.554912    2572 projected.go:196] Error preparing data for projected volume kube-api-access-b5lhj for pod openshift-network-diagnostics/network-check-target-m76pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:27.555715 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.554953    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj podName:865029c1-7176-4b5c-aa12-f4a237af8e60 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:43.554936755 +0000 UTC m=+34.326924521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b5lhj" (UniqueName: "kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj") pod "network-check-target-m76pn" (UID: "865029c1-7176-4b5c-aa12-f4a237af8e60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:37:27.672855 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.672831    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:27.672950 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.672922    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:27.672950 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.672934    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:27.673049 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.673015    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:27.673182 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.673117    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:27.673602 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:27.673297    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:27.819535 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.819500    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"1ad2b49f87d691b9507bbc0c86129b40eed2c35af791fc578506ef01f3c3a098"}
Nov 03 18:37:27.819683 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.819571    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"2aa1f0be005843cd3d962843e072c2c561b902e3bdeecdbd56e42ccc87752a39"}
Nov 03 18:37:27.819683 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.819615    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"3a7b9c9a48c94ce529db31f8b6a096f76e3c0ddeeb667dd8f3f4cd2c59091706"}
Nov 03 18:37:27.819683 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.819644    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"742107023a3309ad2389828619c816890c119d1655e2668e84dad56e558d442b"}
Nov 03 18:37:27.819683 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.819672    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"15533686cac6069c2628b657ee8d1f4861056596f731ad30f5dd4f25c542b2b8"}
Nov 03 18:37:27.819857 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:27.819697    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"13d9761750427b1846385451e24de9b5d1bdd01e3acbefadf5461803ed260f16"}
Nov 03 18:37:28.824504 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:28.824313    2572 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="6886d5d0a6f3b1d6445c44a978b008b8fed0a01db105d56561f9f485b5335d77" exitCode=0
Nov 03 18:37:28.824504 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:28.824385    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerDied","Data":"6886d5d0a6f3b1d6445c44a978b008b8fed0a01db105d56561f9f485b5335d77"}
Nov 03 18:37:29.673962 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:29.673927    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:29.673962 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:29.673949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:29.674262 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:29.674231    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:29.674680 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:29.674662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:29.674822 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:29.674802    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:29.674937 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:29.674914    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:29.828966 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:29.828940    2572 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="41775b54ac345144cca43f96b9d468b1ebcada8ef4c0123815f70b8435042a06" exitCode=0
Nov 03 18:37:29.829401 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:29.829003    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerDied","Data":"41775b54ac345144cca43f96b9d468b1ebcada8ef4c0123815f70b8435042a06"}
Nov 03 18:37:30.834055 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:30.834020    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lx772" event={"ID":"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866","Type":"ContainerStarted","Data":"777f17bdf76e25fb86065e39b6dbb588b7315bcab8a9b3993dd1643b39733194"}
Nov 03 18:37:30.836998 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:30.836971    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"14aacd1efa66bd1f4c6ca1cc361fb3bb86525053635acbe74459ccdc0e0c0079"}
Nov 03 18:37:30.957558 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:30.957509    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lx772" podStartSLOduration=7.530121319 podStartE2EDuration="20.957496943s" totalImagesPullingTime="13.427375624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:30.956945495 +0000 UTC m=+21.728933280" watchObservedRunningTime="2026-11-03 18:37:30.957496943 +0000 UTC m=+21.729484729"
Nov 03 18:37:31.675712 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:31.675682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:31.675877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:31.675683    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:31.675877 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:31.675781    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:31.675877 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:31.675853    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:31.675877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:31.675690    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:31.676013 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:31.675920    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:32.844527 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:32.844496    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" event={"ID":"8903afd0-dd9e-42e1-9aa0-d228f3ef611c","Type":"ContainerStarted","Data":"5a1392228db6da444f64061def1d2c4dc3b3eef7a02f5ae07b7476e812765595"}
Nov 03 18:37:32.845303 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:32.844794    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:32.845303 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:32.844878    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:32.860727 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:32.860583    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:32.860855 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:32.860799    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:33.128852 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:33.128709    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb" podStartSLOduration=7.806619665 podStartE2EDuration="23.128696279s" totalImagesPullingTime="15.322076614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:33.12848734 +0000 UTC m=+23.900475128" watchObservedRunningTime="2026-11-03 18:37:33.128696279 +0000 UTC m=+23.900684064"
Nov 03 18:37:33.672864 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:33.672821    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:33.672864 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:33.672843    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:33.673086 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:33.672972    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:33.673086 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:33.673009    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:33.673217 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:33.673122    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:33.673277 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:33.673233    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:33.846834 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:33.846809    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:34.294652 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.294620    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:34.294828 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.294745    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:37:34.295241 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.295221    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:34.536769 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.536698    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xr6hd"]
Nov 03 18:37:34.536906 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.536798    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:34.536946 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:34.536907    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:34.540217 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.540192    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m76pn"]
Nov 03 18:37:34.540326 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.540275    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:34.540373 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:34.540344    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:34.544131 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.544111    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7sqm4"]
Nov 03 18:37:34.544241 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:34.544206    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:34.544285 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:34.544274    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:35.672964 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:35.672922    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:35.673402 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:35.673034    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:36.672508 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:36.672477    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:36.672654 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:36.672584    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:36.672654 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:36.672634    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:36.672756 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:36.672737    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:37.673509 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:37.673478    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:37.673916 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:37.673584    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m76pn" podUID="865029c1-7176-4b5c-aa12-f4a237af8e60"
Nov 03 18:37:38.672853 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:38.672828    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:38.672998 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:38.672924    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xr6hd" podUID="793f3faa-8090-4008-a15c-13202c13e875"
Nov 03 18:37:38.673061 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:38.673017    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:38.673181 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:38.673161    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7sqm4" podUID="1870fed7-afe2-43c7-ad6d-be96cf7624f2"
Nov 03 18:37:39.441457 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.441429    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeReady"
Nov 03 18:37:39.441878 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.441575    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 18:37:39.557489 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.557461    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-khjmx"]
Nov 03 18:37:39.559218 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.559203    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9pxqg"]
Nov 03 18:37:39.559337 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.559320    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.560712 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.560694    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:39.621695 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.621644    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 18:37:39.621695 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.621649    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 18:37:39.622190 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.622164    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 18:37:39.622284 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.622246    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc7sj\""
Nov 03 18:37:39.626680 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.626659    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9pxqg"]
Nov 03 18:37:39.626786 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.626756    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zjfk4\""
Nov 03 18:37:39.646060 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.646040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj78x\" (UniqueName: \"kubernetes.io/projected/93974d9c-7159-4b45-8bc7-184f9534f270-kube-api-access-lj78x\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:39.646172 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.646157    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.646216 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.646191    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93974d9c-7159-4b45-8bc7-184f9534f270-cert\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:39.646248 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.646215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-config-volume\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.646283 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.646248    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-tmp-dir\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.646325 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.646303    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdp6\" (UniqueName: \"kubernetes.io/projected/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-kube-api-access-5mdp6\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.672862 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.672845    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khjmx"]
Nov 03 18:37:39.673058 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.673036    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:39.722637 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.722485    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 18:37:39.722727 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.722522    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 18:37:39.726036 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.726021    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 18:37:39.726221 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.726210    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4grw\""
Nov 03 18:37:39.726468 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.726456    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 18:37:39.746911 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.746890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-config-volume\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.746990 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.746915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-tmp-dir\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.746990 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.746932    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdp6\" (UniqueName: \"kubernetes.io/projected/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-kube-api-access-5mdp6\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.746990 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.746961    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj78x\" (UniqueName: \"kubernetes.io/projected/93974d9c-7159-4b45-8bc7-184f9534f270-kube-api-access-lj78x\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:39.747131 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.747032    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.747131 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.747065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93974d9c-7159-4b45-8bc7-184f9534f270-cert\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:39.747252 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:39.747180    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:39.747252 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:39.747244    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.247227828 +0000 UTC m=+31.019215597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:39.747465 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.747449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-tmp-dir\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.747736 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.747715    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-config-volume\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.750994 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.750972    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93974d9c-7159-4b45-8bc7-184f9534f270-cert\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:39.810736 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.810709    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj78x\" (UniqueName: \"kubernetes.io/projected/93974d9c-7159-4b45-8bc7-184f9534f270-kube-api-access-lj78x\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:39.810833 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.810752    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdp6\" (UniqueName: \"kubernetes.io/projected/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-kube-api-access-5mdp6\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:39.875796 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:39.875741    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:40.248672 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.248637    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9pxqg"]
Nov 03 18:37:40.248777 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:40.248726    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93974d9c_7159_4b45_8bc7_184f9534f270.slice/crio-8a31d0032b9a88f5c9fa9444443449aba26d5eb3ba1c4f6f3984d739eb50f7d6 WatchSource:0}: Error finding container 8a31d0032b9a88f5c9fa9444443449aba26d5eb3ba1c4f6f3984d739eb50f7d6: Status 404 returned error can't find the container with id 8a31d0032b9a88f5c9fa9444443449aba26d5eb3ba1c4f6f3984d739eb50f7d6
Nov 03 18:37:40.250451 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.250434    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:40.250566 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:40.250552    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:40.250613 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:40.250604    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:41.250591208 +0000 UTC m=+32.022578973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:40.672877 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.672851    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:40.673580 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.672893    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:40.726244 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.726220    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 18:37:40.726361 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.726271    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkfh8\""
Nov 03 18:37:40.726361 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.726304    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dvl9\""
Nov 03 18:37:40.729058 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.726701    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 18:37:40.861529 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:40.861503    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9pxqg" event={"ID":"93974d9c-7159-4b45-8bc7-184f9534f270","Type":"ContainerStarted","Data":"8a31d0032b9a88f5c9fa9444443449aba26d5eb3ba1c4f6f3984d739eb50f7d6"}
Nov 03 18:37:41.258652 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:41.258616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:41.258837 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:41.258790    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:41.258899 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:41.258870    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:43.258849896 +0000 UTC m=+34.030837663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:43.272595 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.272558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:43.273272 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:43.272675    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:43.273272 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:43.272741    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:47.27272797 +0000 UTC m=+38.044715750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:43.474429 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.474391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:43.474700 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:43.474507    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 18:37:43.474700 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:43.474571    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:15.474556673 +0000 UTC m=+66.246544437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : secret "metrics-daemon-secret" not found
Nov 03 18:37:43.575508 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.575432    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:43.575508 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.575474    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:43.578263 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.578236    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:43.578498 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.578479    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:43.583727 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.583709    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:43.684482 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.684461    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:43.870100 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.870036    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9pxqg" event={"ID":"93974d9c-7159-4b45-8bc7-184f9534f270","Type":"ContainerStarted","Data":"d99d1de9b7222cafd3ef9e145c27e87ab4fcb4ff347cac3e715ce9a2fa4f6135"}
Nov 03 18:37:43.948475 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:43.948451    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865029c1_7176_4b5c_aa12_f4a237af8e60.slice/crio-91621310a47a0b702dd2f09c18c82e07a8a8a198c6db8ae994c8ffe6bcdc4ea5 WatchSource:0}: Error finding container 91621310a47a0b702dd2f09c18c82e07a8a8a198c6db8ae994c8ffe6bcdc4ea5: Status 404 returned error can't find the container with id 91621310a47a0b702dd2f09c18c82e07a8a8a198c6db8ae994c8ffe6bcdc4ea5
Nov 03 18:37:43.973998 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:43.973975    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m76pn"]
Nov 03 18:37:44.052209 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:44.052136    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793f3faa_8090_4008_a15c_13202c13e875.slice/crio-15953269b153210079beb81d2819bdd289e4765e7754dbfed1dc768cd90fe632 WatchSource:0}: Error finding container 15953269b153210079beb81d2819bdd289e4765e7754dbfed1dc768cd90fe632: Status 404 returned error can't find the container with id 15953269b153210079beb81d2819bdd289e4765e7754dbfed1dc768cd90fe632
Nov 03 18:37:44.052779 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:44.052733    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9pxqg" podStartSLOduration=1.985898088 podStartE2EDuration="5.05271918s" totalImagesPullingTime="3.066821092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:43.998805314 +0000 UTC m=+34.770793110" watchObservedRunningTime="2026-11-03 18:37:44.05271918 +0000 UTC m=+34.824706966"
Nov 03 18:37:44.052900 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:44.052847    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xr6hd"]
Nov 03 18:37:44.873709 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:44.873669    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xr6hd" event={"ID":"793f3faa-8090-4008-a15c-13202c13e875","Type":"ContainerStarted","Data":"15953269b153210079beb81d2819bdd289e4765e7754dbfed1dc768cd90fe632"}
Nov 03 18:37:44.874752 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:44.874732    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m76pn" event={"ID":"865029c1-7176-4b5c-aa12-f4a237af8e60","Type":"ContainerStarted","Data":"91621310a47a0b702dd2f09c18c82e07a8a8a198c6db8ae994c8ffe6bcdc4ea5"}
Nov 03 18:37:46.681764 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.681718    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n9565"]
Nov 03 18:37:46.698330 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.698302    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.741759 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.741675    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n9565"]
Nov 03 18:37:46.790599 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.790497    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 18:37:46.790599 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.790515    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 18:37:46.790599 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.790509    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 18:37:46.790867 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.790649    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 18:37:46.790867 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.790743    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88hbk\""
Nov 03 18:37:46.798937 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.798898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbwqq\" (UniqueName: \"kubernetes.io/projected/ffdce77a-9274-4215-9077-8281171f3c7d-kube-api-access-xbwqq\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.799068 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.798949    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.799068 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.798988    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdce77a-9274-4215-9077-8281171f3c7d-crio-socket\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.799204 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.799118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdce77a-9274-4215-9077-8281171f3c7d-data-volume\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.799204 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.799174    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.900212 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.900173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbwqq\" (UniqueName: \"kubernetes.io/projected/ffdce77a-9274-4215-9077-8281171f3c7d-kube-api-access-xbwqq\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.900384 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.900230    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.900384 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.900269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdce77a-9274-4215-9077-8281171f3c7d-crio-socket\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.900384 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.900319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdce77a-9274-4215-9077-8281171f3c7d-data-volume\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.900384 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.900335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.900384 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:46.900378    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:46.900635 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:46.900398    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 18:37:46.900635 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:46.900445    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:47.40042452 +0000 UTC m=+38.172412285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:46.900635 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:46.900468    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:47.400457283 +0000 UTC m=+38.172445050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : configmap "kube-rbac-proxy" not found
Nov 03 18:37:46.900635 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.900562    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdce77a-9274-4215-9077-8281171f3c7d-crio-socket\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.900845 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.900695    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdce77a-9274-4215-9077-8281171f3c7d-data-volume\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.919898 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.919868    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4tm2v-tn8v7"]
Nov 03 18:37:46.947031 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.946969    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:46.964450 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.964424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbwqq\" (UniqueName: \"kubernetes.io/projected/ffdce77a-9274-4215-9077-8281171f3c7d-kube-api-access-xbwqq\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:46.979428 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:46.979021    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4tm2v-tn8v7"]
Nov 03 18:37:47.004572 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.004534    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2q9jp\""
Nov 03 18:37:47.043698 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.043666    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"]
Nov 03 18:37:47.043990 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.043968    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 18:37:47.062854 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.062833    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"
Nov 03 18:37:47.102481 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.102454    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdh6\" (UniqueName: \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.102622 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.102508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.102622 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.102531    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.102703 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.102619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.105233 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.105195    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 18:37:47.106961 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.106938    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"]
Nov 03 18:37:47.118794 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.118772    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bgcqc\""
Nov 03 18:37:47.143867 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.143835    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 18:37:47.143976 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.143945    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 18:37:47.203268 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.203195    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.203369 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.203267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdh6\" (UniqueName: \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.203369 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.203306    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwcp7\" (UniqueName: \"kubernetes.io/projected/ad953721-ce75-4e69-a21c-02302526a880-kube-api-access-lwcp7\") pod \"migrator-66df5d496b-78tkg\" (UID: \"ad953721-ce75-4e69-a21c-02302526a880\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"
Nov 03 18:37:47.203369 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.203334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.203369 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.203359    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.203990 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.203969    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.204184 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.204135    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.204276 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.204264    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.264994 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.264963    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdh6\" (UniqueName: \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.276760 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.276733    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:47.307194 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.303916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:47.307194 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.303976    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwcp7\" (UniqueName: \"kubernetes.io/projected/ad953721-ce75-4e69-a21c-02302526a880-kube-api-access-lwcp7\") pod \"migrator-66df5d496b-78tkg\" (UID: \"ad953721-ce75-4e69-a21c-02302526a880\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"
Nov 03 18:37:47.307194 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:47.304061    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:47.307194 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:47.304134    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.304113642 +0000 UTC m=+46.076101419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:47.366494 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.366467    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwcp7\" (UniqueName: \"kubernetes.io/projected/ad953721-ce75-4e69-a21c-02302526a880-kube-api-access-lwcp7\") pod \"migrator-66df5d496b-78tkg\" (UID: \"ad953721-ce75-4e69-a21c-02302526a880\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"
Nov 03 18:37:47.389434 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.389407    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"
Nov 03 18:37:47.404745 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.404722    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:47.404881 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.404792    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:47.404936 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:47.404882    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:47.404936 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:47.404897    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 18:37:47.405005 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:47.404950    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:48.404931529 +0000 UTC m=+39.176919297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:47.405005 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:47.404969    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:48.404959579 +0000 UTC m=+39.176947347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : configmap "kube-rbac-proxy" not found
Nov 03 18:37:47.656925 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:47.656897    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141aa4fe_227d_45f5_a995_1b20655868fd.slice/crio-b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1 WatchSource:0}: Error finding container b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1: Status 404 returned error can't find the container with id b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1
Nov 03 18:37:47.657694 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.657667    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4tm2v-tn8v7"]
Nov 03 18:37:47.786557 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.786532    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"]
Nov 03 18:37:47.883694 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:47.883666    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7" event={"ID":"141aa4fe-227d-45f5-a995-1b20655868fd","Type":"ContainerStarted","Data":"b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1"}
Nov 03 18:37:47.937402 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:47.937373    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad953721_ce75_4e69_a21c_02302526a880.slice/crio-2e6d6ddb748beae58a44a90fb5b1db49e953e1e81b3d5970d140895748e3a861 WatchSource:0}: Error finding container 2e6d6ddb748beae58a44a90fb5b1db49e953e1e81b3d5970d140895748e3a861: Status 404 returned error can't find the container with id 2e6d6ddb748beae58a44a90fb5b1db49e953e1e81b3d5970d140895748e3a861
Nov 03 18:37:48.087317 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.087289    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-69t5n"]
Nov 03 18:37:48.108869 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.108844    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.145257 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.145229    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-69t5n"]
Nov 03 18:37:48.161870 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.161846    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 18:37:48.161966 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.161880    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-442ps\""
Nov 03 18:37:48.190466 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.190441    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 18:37:48.190583 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.190481    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 18:37:48.190583 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.190511    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 18:37:48.211963 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.211929    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-key\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.212099 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.212078    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-cabundle\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.212176 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.212133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbj5x\" (UniqueName: \"kubernetes.io/projected/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-kube-api-access-tbj5x\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.313327 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.313241    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-key\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.313327 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.313313    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-cabundle\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.313527 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.313347    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbj5x\" (UniqueName: \"kubernetes.io/projected/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-kube-api-access-tbj5x\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.314131 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.314065    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-cabundle\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.316008 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.315983    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-key\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.375753 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.375703    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbj5x\" (UniqueName: \"kubernetes.io/projected/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-kube-api-access-tbj5x\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.414392 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.414355    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:48.414516 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.414450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:48.414592 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:48.414578    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:48.414668 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:48.414650    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:50.414633051 +0000 UTC m=+41.186620819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:48.414902 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.414881    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:48.433382 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.433361    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:48.817007 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.816977    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-69t5n"]
Nov 03 18:37:48.817413 ip-10-0-143-45 kubenswrapper[2572]: W1103 18:37:48.817077    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fce30d6_7869_4ce9_ab2a_7c100a2c4f0e.slice/crio-34f53ad17494e2654a8b46058a737eaf6bc0d65b641cb2441a90528c06f8eeea WatchSource:0}: Error finding container 34f53ad17494e2654a8b46058a737eaf6bc0d65b641cb2441a90528c06f8eeea: Status 404 returned error can't find the container with id 34f53ad17494e2654a8b46058a737eaf6bc0d65b641cb2441a90528c06f8eeea
Nov 03 18:37:48.888411 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.888371    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n" event={"ID":"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e","Type":"ContainerStarted","Data":"34f53ad17494e2654a8b46058a737eaf6bc0d65b641cb2441a90528c06f8eeea"}
Nov 03 18:37:48.889652 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.889625    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg" event={"ID":"ad953721-ce75-4e69-a21c-02302526a880","Type":"ContainerStarted","Data":"2e6d6ddb748beae58a44a90fb5b1db49e953e1e81b3d5970d140895748e3a861"}
Nov 03 18:37:48.891620 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.891597    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m76pn" event={"ID":"865029c1-7176-4b5c-aa12-f4a237af8e60","Type":"ContainerStarted","Data":"efefa65f559cb1f730fab7f1e6be392adcf33083a3b034fc4b9f32337847a935"}
Nov 03 18:37:48.891822 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:48.891804    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:49.024321 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:49.024262    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m76pn" podStartSLOduration=35.003058049 podStartE2EDuration="39.024243845s" totalImagesPullingTime="4.021185796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:49.022503516 +0000 UTC m=+39.794491301" watchObservedRunningTime="2026-11-03 18:37:49.024243845 +0000 UTC m=+39.796231633"
Nov 03 18:37:50.428357 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:50.428324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:50.428839 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:50.428440    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:50.428839 ip-10-0-143-45 kubenswrapper[2572]: E1103 18:37:50.428499    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:54.428485447 +0000 UTC m=+45.200473211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:50.798003 ip-10-0-143-45 kubenswrapper[2572]: I1103 18:37:50.797866    2572 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:37:50.798029 ip-10-0-143-45 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 18:37:50.808545 ip-10-0-143-45 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 18:37:50.808710 ip-10-0-143-45 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 18:37:50.808775 ip-10-0-143-45 systemd[1]: kubelet.service: Consumed 2.306s CPU time.
Nov 03 18:37:50.821674 ip-10-0-143-45 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 18:37:50.892307 ip-10-0-143-45 kubenswrapper[4489]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:37:50.892307 ip-10-0-143-45 kubenswrapper[4489]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 18:37:50.892307 ip-10-0-143-45 kubenswrapper[4489]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:37:50.892307 ip-10-0-143-45 kubenswrapper[4489]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:37:50.894371 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894300    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.894371 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894370    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.894371 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894374    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894379    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894383    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894387    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894390    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894394    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894396    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894399    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894401    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894404    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894406    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894409    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894411    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894413    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894416    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894419    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894421    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894423    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894426    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.894468 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894434    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894437    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894440    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894443    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894445    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894448    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894450    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894453    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894455    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894458    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894460    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894463    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894466    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894468    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894471    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894474    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894477    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894479    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894482    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894484    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.894966 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894487    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894490    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894492    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894495    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894497    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894500    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894503    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894505    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894509    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894511    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894514    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894517    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894519    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894522    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894525    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894528    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894530    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894533    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894535    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.895479 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894537    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894540    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894542    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894545    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894548    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894551    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894554    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894556    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894559    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894561    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894564    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894567    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894569    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894572    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894574    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894576    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894579    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894583    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894587    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.895942 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894589    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894592    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894594    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894597    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894599    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894601    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894604    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894606    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894610    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894612    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894615    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894617    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894620    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894622    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894624    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894627    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894629    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894632    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894634    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894637    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.896411 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894639    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894642    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894644    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894647    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894649    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894652    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.894654    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895411    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895418    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895421    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895424    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895427    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895429    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895431    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895434    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895437    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895440    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895442    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895445    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.896887 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895447    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895450    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895453    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895456    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895458    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895461    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895463    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895466    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895468    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895471    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895474    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895476    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895479    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895482    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895484    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895487    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895490    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895492    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895495    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.897370 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895498    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895500    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895503    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895505    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895523    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895528    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895534    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895539    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895543    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895547    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895551    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895555    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895559    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895563    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895568    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895571    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895573    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895576    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895578    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895581    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.897829 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895583    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895585    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895590    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895593    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895597    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895600    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895602    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895605    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895607    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895610    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895613    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895615    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895618    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895621    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895624    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895626    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895629    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895632    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895634    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.898330 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895637    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895640    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895642    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895645    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895647    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895649    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895652    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895655    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895657    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895660    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895662    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895665    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895667    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895669    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895672    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895674    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895677    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895680    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895683    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.898778 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895685    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895689    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895693    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895695    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895698    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895700    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895703    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895706    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895708    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895711    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895713    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895716    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895718    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895720    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895723    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895725    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.895728    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896256    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896263    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.899241 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896267    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896270    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896273    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896276    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896279    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896281    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896284    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896286    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896289    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896292    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896295    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896297    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896300    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896302    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896305    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896308    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896310    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896313    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896315    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.899693 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896318    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896322    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896324    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896327    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896330    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896332    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896335    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896338    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896340    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896343    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896345    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896347    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896350    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896352    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896355    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896358    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896361    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896364    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896366    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896369    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.900218 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896372    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896374    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896376    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896379    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896383    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896386    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896389    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896392    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896395    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896398    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896401    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896403    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896406    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896409    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896412    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896415    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896418    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896420    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896423    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.900697 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896425    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896428    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896431    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896434    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896439    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896441    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896444    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896446    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896449    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896452    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896454    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896457    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896459    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896462    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896464    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896466    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896469    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896471    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896474    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.901449 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896478    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896482    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896485    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896488    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896490    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896493    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896495    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896498    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896500    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896503    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896505    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896508    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896510    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896513    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896515    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896517    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896520    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896524    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896530    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896532    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.902214 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896535    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896537    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896540    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896542    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896544    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896548    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.896551    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896638    4489 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896646    4489 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896651    4489 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896656    4489 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896660    4489 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896663    4489 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896667    4489 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896671    4489 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896674    4489 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896677    4489 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896681    4489 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896684    4489 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896687    4489 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896690    4489 flags.go:64] FLAG: --cgroup-root=""
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896693    4489 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896696    4489 flags.go:64] FLAG: --client-ca-file=""
Nov 03 18:37:50.902756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896699    4489 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896702    4489 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896707    4489 flags.go:64] FLAG: --cluster-domain=""
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896709    4489 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896712    4489 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896716    4489 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896720    4489 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896726    4489 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896728    4489 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896731    4489 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896734    4489 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896737    4489 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896740    4489 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896743    4489 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896746    4489 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896749    4489 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896754    4489 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896757    4489 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896760    4489 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896762    4489 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896765    4489 flags.go:64] FLAG: --enable-server="true"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896768    4489 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896772    4489 flags.go:64] FLAG: --event-burst="100"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896775    4489 flags.go:64] FLAG: --event-qps="50"
Nov 03 18:37:50.903410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896777    4489 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896780    4489 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896783    4489 flags.go:64] FLAG: --eviction-hard=""
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896787    4489 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896790    4489 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896793    4489 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896796    4489 flags.go:64] FLAG: --eviction-soft=""
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896799    4489 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896802    4489 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896805    4489 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896808    4489 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896811    4489 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896813    4489 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896816    4489 flags.go:64] FLAG: --feature-gates=""
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896821    4489 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896826    4489 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896829    4489 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896832    4489 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896835    4489 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896838    4489 flags.go:64] FLAG: --help="false"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896840    4489 flags.go:64] FLAG: --hostname-override="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896843    4489 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896846    4489 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 18:37:50.904066 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896849    4489 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896853    4489 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896856    4489 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896859    4489 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896862    4489 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896865    4489 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896867    4489 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896870    4489 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896873    4489 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896876    4489 flags.go:64] FLAG: --kube-reserved=""
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896879    4489 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896881    4489 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896884    4489 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896888    4489 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896891    4489 flags.go:64] FLAG: --lock-file=""
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896894    4489 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896897    4489 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896900    4489 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896906    4489 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896908    4489 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896911    4489 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896914    4489 flags.go:64] FLAG: --logging-format="text"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896917    4489 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896921    4489 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 18:37:50.904635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896925    4489 flags.go:64] FLAG: --manifest-url=""
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896928    4489 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896932    4489 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896950    4489 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896955    4489 flags.go:64] FLAG: --max-pods="110"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896958    4489 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896962    4489 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896965    4489 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896968    4489 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896971    4489 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896974    4489 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896976    4489 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896983    4489 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896986    4489 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.896989    4489 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897003    4489 flags.go:64] FLAG: --pod-cidr=""
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897006    4489 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897009    4489 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897012    4489 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897015    4489 flags.go:64] FLAG: --port="10250"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897018    4489 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897021    4489 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0bccba2f5c2a95c50"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897024    4489 flags.go:64] FLAG: --qos-reserved=""
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897027    4489 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897030    4489 flags.go:64] FLAG: --register-node="true"
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897033    4489 flags.go:64] FLAG: --register-with-taints=""
Nov 03 18:37:50.905331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897036    4489 flags.go:64] FLAG: --registry-burst="10"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897039    4489 flags.go:64] FLAG: --registry-qps="5"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897042    4489 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897045    4489 flags.go:64] FLAG: --reserved-memory=""
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897049    4489 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897052    4489 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897056    4489 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897059    4489 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897062    4489 flags.go:64] FLAG: --runonce="false"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897064    4489 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897067    4489 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897070    4489 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897073    4489 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897076    4489 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897079    4489 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897082    4489 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897085    4489 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897087    4489 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897090    4489 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897093    4489 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897096    4489 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897099    4489 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897101    4489 flags.go:64] FLAG: --system-cgroups=""
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897104    4489 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897110    4489 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 18:37:50.906089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897113    4489 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897116    4489 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897120    4489 flags.go:64] FLAG: --tls-min-version=""
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897122    4489 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897125    4489 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897128    4489 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897131    4489 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897134    4489 flags.go:64] FLAG: --v="2"
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897152    4489 flags.go:64] FLAG: --version="false"
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897156    4489 flags.go:64] FLAG: --vmodule=""
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897161    4489 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897164    4489 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897330    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897337    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897339    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897342    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897344    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897347    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897349    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897352    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897355    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897357    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.906695 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897360    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897364    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897368    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897371    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897375    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897378    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897381    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897383    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897386    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897389    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897392    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897394    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897396    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897399    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897402    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897404    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897407    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897409    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897412    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.907307 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897414    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897418    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897420    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897423    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897427    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897429    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897432    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897436    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897439    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897442    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897445    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897448    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897450    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897453    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897455    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897458    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897460    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897463    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897465    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.907748 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897468    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897470    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897473    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897475    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897479    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897481    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897484    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897486    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897489    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897491    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897494    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897496    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897499    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897502    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897504    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897513    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897516    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897520    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897523    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897525    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.908212 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897528    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897530    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897533    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897535    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897538    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897540    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897542    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897545    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897548    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897550    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897552    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897555    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897557    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897560    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897562    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897565    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897569    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897571    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897573    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.908679 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897576    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897578    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897581    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897583    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897585    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897588    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897590    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897593    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897595    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897603    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897608    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897611    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897613    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897616    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897618    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897621    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897623    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897625    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.909134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897628    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.897654    4489 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897703    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897707    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897710    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897713    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897716    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897718    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897721    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897723    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897726    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897729    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897732    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897744    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897751    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897754    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.909579 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897757    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897759    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897762    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897764    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897767    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897769    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897772    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897774    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897782    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897786    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897789    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897791    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897794    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897796    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897799    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897801    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897804    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897806    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897809    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.909971 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897811    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897814    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897816    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897818    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897821    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897823    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897826    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897828    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897832    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897835    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897838    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897840    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897843    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897845    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897848    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897850    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897853    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897855    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897858    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897860    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.910440 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897863    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897865    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897874    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897878    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897880    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897882    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897885    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897887    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897890    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897892    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897895    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897897    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897901    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897904    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897908    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897911    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897913    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897916    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897918    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897921    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.910919 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897924    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897927    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897929    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897932    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897934    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897937    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897947    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897949    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897952    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897954    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897957    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897959    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897962    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897964    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897968    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897976    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897979    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897982    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897986    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.911506 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897989    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897992    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897995    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.897998    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898002    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898005    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898007    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898010    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898012    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898014    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898017    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898019    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898022    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.898025    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.903092    4489 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.903108    4489 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.903131    4489 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.903136    4489 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903213    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903219    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.911953 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903222    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903225    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903228    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903230    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903233    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903236    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903239    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903242    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903244    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903247    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903249    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903252    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903255    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903259    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903263    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903266    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903269    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903272    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903275    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.912447 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903278    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903281    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903283    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903286    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903288    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903291    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903294    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903301    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903304    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903306    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903309    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903311    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903314    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903316    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903319    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903321    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903324    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903326    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903329    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.912991 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903331    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903334    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903336    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903339    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903342    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903344    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903347    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903350    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903352    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903355    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903358    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903361    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903363    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903366    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903368    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903371    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903373    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903377    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903381    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903384    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.913532 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903387    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903389    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903392    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903394    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903397    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903399    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903402    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903404    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903407    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903409    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903412    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903414    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903418    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903420    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903423    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903426    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903428    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903432    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903434    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903437    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.914134 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903440    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903442    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903445    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903448    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903450    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903453    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903455    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903457    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903460    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903462    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903465    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903467    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903470    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903472    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903475    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903477    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903479    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903482    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903484    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.914710 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903487    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903489    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903492    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903494    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903496    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903499    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903502    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.903529    4489 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903664    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903669    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903672    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903674    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903677    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903680    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903683    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903685    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.915228 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903688    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903690    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903693    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903695    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903698    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903700    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903703    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903705    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903707    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903710    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903712    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903714    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903717    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903719    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903722    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903724    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903726    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903729    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903731    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.915606 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903733    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903736    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903754    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903759    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903761    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903764    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903767    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903770    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903772    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903775    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903777    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903780    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903783    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903786    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903788    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903791    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903793    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903796    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903798    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903801    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.916046 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903803    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903806    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903808    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903810    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903813    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903815    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903818    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903820    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903823    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903825    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903828    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903830    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903833    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903835    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903838    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903840    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903843    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903845    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903848    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903851    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.916555 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903853    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903856    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903858    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903861    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903864    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903867    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903869    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903872    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903874    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903877    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903879    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903881    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903884    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903886    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903889    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903891    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903893    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903896    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903898    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903901    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.917016 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903903    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903905    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903908    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903910    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903913    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903915    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903919    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903923    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903926    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903929    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903932    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903936    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903939    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903941    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903944    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903947    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903951    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903954    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.917525 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.903957    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.903980    4489 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904010    4489 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904015    4489 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904019    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904022    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904024    4489 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904027    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904029    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904032    4489 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904034    4489 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904038    4489 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904042    4489 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904046    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904049    4489 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:37:50.917956 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904052    4489 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904055    4489 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904058    4489 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904060    4489 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904063    4489 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904065    4489 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904068    4489 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904071    4489 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904073    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904076    4489 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904079    4489 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904082    4489 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904084    4489 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904086    4489 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904089    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904091    4489 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904093    4489 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904097    4489 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904099    4489 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:37:50.918327 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904102    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904104    4489 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904106    4489 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904109    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904111    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904113    4489 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904116    4489 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904118    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904120    4489 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904123    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904125    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904128    4489 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904130    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904132    4489 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904135    4489 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904153    4489 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904157    4489 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904160    4489 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904162    4489 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:37:50.918812 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904165    4489 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904168    4489 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904170    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904173    4489 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904175    4489 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904178    4489 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904181    4489 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904183    4489 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904186    4489 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904188    4489 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904190    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904193    4489 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904196    4489 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904198    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904201    4489 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904203    4489 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904206    4489 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904209    4489 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904211    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904214    4489 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:37:50.919328 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904216    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904219    4489 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904221    4489 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904226    4489 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904228    4489 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904231    4489 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904233    4489 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904235    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904238    4489 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904240    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904244    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904246    4489 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904249    4489 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904251    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904253    4489 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904256    4489 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904259    4489 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904261    4489 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904264    4489 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904266    4489 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:37:50.919779 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904269    4489 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904271    4489 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904274    4489 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904276    4489 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904279    4489 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904282    4489 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904285    4489 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904288    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904290    4489 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904293    4489 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904295    4489 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904298    4489 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904300    4489 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904303    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:37:50.904305    4489 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.904358    4489 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.905275    4489 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.905327    4489 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.905498    4489 server.go:1014] "Starting client certificate rotation"
Nov 03 18:37:50.920258 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.905653    4489 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 18:37:50.920701 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.905711    4489 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:32:09 +0000 UTC" deadline="2028-06-28 00:22:34.070444984 +0000 UTC"
Nov 03 18:37:50.920701 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.905760    4489 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14453h44m43.164687896s"
Nov 03 18:37:50.920701 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.906022    4489 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:37:50.922494 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.922478    4489 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:37:50.923804 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.923790    4489 log.go:25] "Validated CRI v1 runtime API"
Nov 03 18:37:50.925324 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.925310    4489 log.go:25] "Validated CRI v1 image API"
Nov 03 18:37:50.926030 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.926014    4489 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 18:37:50.927639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.927613    4489 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a00575e-a0e5-421b-a045-6c7c66e9f3dd:/dev/nvme0n1p3 a9b9e9a1-f123-47f2-85d3-ebb4d20e42bd:/dev/nvme0n1p4]
Nov 03 18:37:50.927867 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.927636    4489 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/017f22f4a10e222aacb30b63536e86e280461b31fb3664381eba47513819142d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/017f22f4a10e222aacb30b63536e86e280461b31fb3664381eba47513819142d/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15953269b153210079beb81d2819bdd289e4765e7754dbfed1dc768cd90fe632/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15953269b153210079beb81d2819bdd289e4765e7754dbfed1dc768cd90fe632/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16cd488feffbe269e60eda61387381df8760e22e4cdb49dee7ed3cd507393082/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16cd488feffbe269e60eda61387381df8760e22e4cdb49dee7ed3cd507393082/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22b221e79df93b7ed2a9c787d47d152fce8fdd2438ac168e72ebe9eedaa29e92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22b221e79df93b7ed2a9c787d47d152fce8fdd2438ac168e72ebe9eedaa29e92/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e6d6ddb748beae58a44a90fb5b1db49e953e1e81b3d5970d140895748e3a861/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e6d6ddb748beae58a44a90fb5b1db49e953e1e81b3d5970d140895748e3a861/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34f53ad17494e2654a8b46058a737eaf6bc0d65b641cb2441a90528c06f8eeea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34f53ad17494e2654a8b46058a737eaf6bc0d65b641cb2441a90528c06f8eeea/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a31d0032b9a88f5c9fa9444443449aba26d5eb3ba1c4f6f3984d739eb50f7d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a31d0032b9a88f5c9fa9444443449aba26d5eb3ba1c4f6f3984d739eb50f7d6/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91621310a47a0b702dd2f09c18c82e07a8a8a198c6db8ae994c8ffe6bcdc4ea5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91621310a47a0b702dd2f09c18c82e07a8a8a198c6db8ae994c8ffe6bcdc4ea5/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5b903e852e4a4671b4df9924549604a77a090a568fd3ab228e9fb6c420cb3be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5b903e852e4a4671b4df9924549604a77a090a568fd3ab228e9fb6c420cb3be/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee53dcab1461cda6f09acdcf9dc0e732fb3f5f9340d29af4ab95cd5867f0bed0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee53dcab1461cda6f09acdcf9dc0e732fb3f5f9340d29af4ab95cd5867f0bed0/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4b07bde72d02f008a095963395e92a5955bc92700c7df515365c015a643a57e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4b07bde72d02f008a095963395e92a5955bc92700c7df515365c015a643a57e/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb8ab53b2c67e8cefe666b75b67a16233bfdbe811c713222352180690b2cf3d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb8ab53b2c67e8cefe666b75b67a16233bfdbe811c713222352180690b2cf3d6/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcad5aec9f095876d6ab9d8524fefed8d0c03b7195a6ce1449c0cf9e3d725a71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcad5aec9f095876d6ab9d8524fefed8d0c03b7195a6ce1449c0cf9e3d725a71/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/038e7f0f-0691-4006-aaee-e8b99231be0a/volumes/kubernetes.io~projected/kube-api-access-dr6b4:{mountpoint:/var/lib/kubelet/pods/038e7f0f-0691-4006-aaee-e8b99231be0a/volumes/kubernetes.io~projected/kube-api-access-dr6b4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141aa4fe-227d-45f5-a995-1b20655868fd/volumes/kubernetes.io~projected/kube-api-access-ftdh6:{mountpoint:/var/lib/kubelet/pods/141aa4fe-227d-45f5-a995-1b20655868fd/volumes/kubernetes.io~projected/kube-api-access-ftdh6 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1870fed7-afe2-43c7-ad6d-be96cf7624f2/volumes/kubernetes.io~projected/kube-api-access-n9fsn:{mountpoint:/var/lib/kubelet/pods/1870fed7-afe2-43c7-ad6d-be96cf7624f2/volumes/kubernetes.io~projected/kube-api-access-n9fsn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e/volumes/kubernetes.io~projected/kube-api-access-tbj5x:{mountpoint:/var/lib/kubelet/pods/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e/volumes/kubernetes.io~projected/kube-api-access-tbj5x major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21ec2464-111d-4061-9938-44d42bea81b7/volumes/kubernetes.io~projected/kube-api-access-4bhgv:{mountpoint:/var/lib/kubelet/pods/21ec2464-111d-4061-9938-44d42bea81b7/volumes/kubernetes.io~projected/kube-api-access-4bhgv major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866/volumes/kubernetes.io~projected/kube-api-access-tv7wh:{mountpoint:/var/lib/kubelet/pods/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866/volumes/kubernetes.io~projected/kube-api-access-tv7wh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/344ba3f2-5bd5-4e78-bd7d-101ad41f998e/volumes/kubernetes.io~projected/kube-api-access-vjwzg:{mountpoint:/var/lib/kubelet/pods/344ba3f2-5bd5-4e78-bd7d-101ad41f998e/volumes/kubernetes.io~projected/kube-api-access-vjwzg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43940503-68c4-41e5-9656-4e1fe7837110/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/43940503-68c4-41e5-9656-4e1fe7837110/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/793f3faa-8090-4008-a15c-13202c13e875/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/793f3faa-8090-4008-a15c-13202c13e875/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/865029c1-7176-4b5c-aa12-f4a237af8e60/volumes/kubernetes.io~projected/kube-api-access-b5lhj:{mountpoint:/var/lib/kubelet/pods/865029c1-7176-4b5c-aa12-f4a237af8e60/volumes/kubernetes.io~projected/kube-api-access-b5lhj major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/878a9a2c-5584-4c23-8b39-754abed5d341/volumes/kubernetes.io~projected/kube-api-access-lltc4:{mountpoint:/var/lib/kubelet/pods/878a9a2c-5584-4c23-8b39-754abed5d341/volumes/kubernetes.io~projected/kube-api-access-lltc4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volumes/kubernetes.io~projected/kube-api-access-jrsrc:{mountpoint:/var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volumes/kubernetes.io~projected/kube-api-access-jrsrc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93974d9c-7159-4b45-8bc7-184f9534f270/volumes/kubernetes.io~projected/kube-api-access-lj78x:{mountpoint:/var/lib/kubelet/pods/93974d9c-7159-4b45-8bc7-184f9534f270/volumes/kubernetes.io~projected/kube-api-access-lj78x major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93974d9c-7159-4b45-8bc7-184f9534f270/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/93974d9c-7159-4b45-8bc7-184f9534f270/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~projected/kube-api-access-dsnmx:{mountpoint:/var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~projected/kube-api-access-dsnmx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f/volumes/kubernetes.io~projected/kube-api-access-5mdp6:{mountpoint:/var/lib/kubelet/pods/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f/volumes/kubernetes.io~projected/kube-api-access-5mdp6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad953721-ce75-4e69-a21c-02302526a880/volumes/kubernetes.io~projected/kube-api-access-lwcp7:{mountpoint:/var/lib/kubelet/pods/ad953721-ce75-4e69-a21c-02302526a880/volumes/kubernetes.io~projected/kube-api-access-lwcp7 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c167fdc3-2279-434d-bb1a-022b46d24b2c/volumes/kubernetes.io~projected/kube-api-access-2pwsl:{mountpoint:/var/lib/kubelet/pods/c167fdc3-2279-434d-bb1a-022b46d24b2c/volumes/kubernetes.io~projected/kube-api-access-2pwsl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffdce77a-9274-4215-9077-8281171f3c7d/volumes/kubernetes.io~projected/kube-api-access-xbwqq:{mountpoint:/var/lib/kubelet/pods/ffdce77a-9274-4215-9077-8281171f3c7d/volumes/kubernetes.io~projected/kube-api-access-xbwqq major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a2b215d03abc77123a5f662966be9f9090cb08afb5a9008598fe0f2193f58620/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ceb3ee91dcea9c20003fa231ab81d528e547910d079bf2150f527a31dfa0d9b1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3533aa709321f183e931af81c9da97e495e4fc115bc5c6934e24395a9060e994/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b3405221e57e43314255c4e829f6649f4741f571c2e11acdcd86f53caebe2f03/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cfe10c339c283bc79fdcfbfc8eaf74a3b670d91c1b655cc8ed56fc6253769d9e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3f20ce76a6095dc03fa8f7cbc310f401857fe221862682752c8b1509d03851d1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/f07091adc9f5f402f6023d2c02ab75c79a99fa92ff249529aec7f6202b3f9e88/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2f6404779a838acc756eb8b0d696c3ddc1312eed47778b330738434f76d5f228/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/f6f33f1b20c66a9113d66b8d874cba96430bd42bced29f2ef149657b0c8f93b8/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1aefb048ebee0af77117e300f1039206ed6a2452757cfe6c6d6ce911b976e4ae/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/09a0b9bc2e8318135d4be90c46c6caa5708a031752dd16d710ea0afeb51d4883/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/4654afef03dc7a03213e52b3d0ba941a10c02b3fd2d95bb455da3409dc3c7fea/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/092e1c5755a7c4d0e8ab51e6d10ed15c3fa9c1bc8eefdb2a79f1fc176e61616d/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/044c2d6fb48e283820ce6e637486db67288ac8e63bb1a7a4e536c664fd9a2c2e/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b769eafb135c4e74026218b0955a7c93707d27362291d2db882023fef43bc087/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/40c942f37c945c7bec1065d8ddad61546bd38fc3121c06eacf4dd7d1d3c199c7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8d49c1fc0aaf14d6c0ecfb731fff409cbc61f2687da0e74f43dae3f7369f6977/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/32a7dc42c4d4e165c2550e5424275dc8afebf49accfbc820d35a88ae67dd04a8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b7738a80987b66dce76264d3ab3889c1cb0471ab74720046e067d1cc7c0ec432/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/728e62c30a8bc28bfde0dfc68ad4433ab0e295558ebd302b3eda2b6e5fd3df6d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e98011eb6b53967eda54dbd9d9d81e2ea9cf5a2b0975e14763272be6d171314f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/11cd471fe8e4a49242cfa69e6aed12bd5965f815ef650a36412b5c2861c751a6/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/b7562ed16ce8c4d7a500b4fc4e1f57e99187f80482f8a7957fec44f45dcea8ef/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/45dfa133b442f7af819d64f5ed75913a2bac92fcd732eec01f0a27c41130b456/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/e324ea9b1dea3964f23d40642c856dcfc1559df39a470f808fb5e5d959e9f824/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/e90d4de6c6f0580a0b29cf3a478942eae3e84a981899a027340d5ad0bd3e29b9/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/2f9b5639a19c05172380362ddc76398054f63ed8b811cb65b1844d527ba68959/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/8dfdeb932b2a4a23d8caa2bbbfe49707ca467ee0257e3a90316d6bb004f0272e/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/28342049f1ab709bf4182d0863e0d6c82a1a70e238c227245f6fb9e27ee61398/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/43f6f133f64c3c3daeb820f8a88a4eec7c3348913e391a480242eaca6325c433/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/7808c61eaf132dbe3857c9715c9d8ef2d808830a6c61ebe425628a9077644b65/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fb94466a61ad4a5177f22ae1b90b53428378826cf3aee31f552bbdbf1fc7dca6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/703f74245d6e35fd45611bf0bc1a1305d0383711cfe8c737cbbd9d152114d4c2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/09f3a7c1e378674c05d90dad627555b5f19b9d4b2fe211ed20303d8bcec972b2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/0e97a36e92a4933761c7ed5149f8b9054828f60adc7b7082cea5a1b6286e30a8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/dd04117e0c23800b2fec8e9c81cfc25a185fe8a1dffe1bb9a27f3c10f9b2ce2e/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e1f81c53fbe5fb8833e6519b349374e03662e4485dea7377ee630969a2225b40/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/49be73a4be2f227658c0cb8e2324dc9b4c0e1a33da182dbe2cb83b161c2d4ff4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8d8ae074845da8a0ba00e73af7fce4288aafd67f028bcd2f08ed1c6027b92b51/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/09686afb0b5c4dd6afb7326d46e21ed97f4b375ed591b9ffed9ecd9551419a91/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/455fe0aa55aecb8d01a61a57a9f96622870c11894b6dd657d39413355d871ccb/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 18:37:50.936258 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.935996    4489 manager.go:217] Machine: {Timestamp:2026-11-03 18:37:50.935540717 +0000 UTC m=+0.084235278 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097752 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f858f5bf8eeefa83a34fbe47d76f4 SystemUUID:ec2f858f-5bf8-eeef-a83a-34fbe47d76f4 BootID:3d217963-b0ef-416b-8f44-2dec473f3f87 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcad5aec9f095876d6ab9d8524fefed8d0c03b7195a6ce1449c0cf9e3d725a71/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/793f3faa-8090-4008-a15c-13202c13e875/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e6d6ddb748beae58a44a90fb5b1db49e953e1e81b3d5970d140895748e3a861/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16cd488feffbe269e60eda61387381df8760e22e4cdb49dee7ed3cd507393082/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4b07bde72d02f008a095963395e92a5955bc92700c7df515365c015a643a57e/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~projected/kube-api-access-dsnmx DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a31d0032b9a88f5c9fa9444443449aba26d5eb3ba1c4f6f3984d739eb50f7d6/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34f53ad17494e2654a8b46058a737eaf6bc0d65b641cb2441a90528c06f8eeea/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb8ab53b2c67e8cefe666b75b67a16233bfdbe811c713222352180690b2cf3d6/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/017f22f4a10e222aacb30b63536e86e280461b31fb3664381eba47513819142d/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93974d9c-7159-4b45-8bc7-184f9534f270/volumes/kubernetes.io~projected/kube-api-access-lj78x DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/865029c1-7176-4b5c-aa12-f4a237af8e60/volumes/kubernetes.io~projected/kube-api-access-b5lhj DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volumes/kubernetes.io~projected/kube-api-access-jrsrc DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c167fdc3-2279-434d-bb1a-022b46d24b2c/volumes/kubernetes.io~projected/kube-api-access-2pwsl DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5b903e852e4a4671b4df9924549604a77a090a568fd3ab228e9fb6c420cb3be/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93974d9c-7159-4b45-8bc7-184f9534f270/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866/volumes/kubernetes.io~projected/kube-api-access-tv7wh DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffdce77a-9274-4215-9077-8281171f3c7d/volumes/kubernetes.io~projected/kube-api-access-xbwqq DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/344ba3f2-5bd5-4e78-bd7d-101ad41f998e/volumes/kubernetes.io~projected/kube-api-access-vjwzg DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/878a9a2c-5584-4c23-8b39-754abed5d341/volumes/kubernetes.io~projected/kube-api-access-lltc4 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/141aa4fe-227d-45f5-a995-1b20655868fd/volumes/kubernetes.io~projected/kube-api-access-ftdh6 DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1870fed7-afe2-43c7-ad6d-be96cf7624f2/volumes/kubernetes.io~projected/kube-api-access-n9fsn DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f/volumes/kubernetes.io~projected/kube-api-access-5mdp6 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/21ec2464-111d-4061-9938-44d42bea81b7/volumes/kubernetes.io~projected/kube-api-access-4bhgv DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/038e7f0f-0691-4006-aaee-e8b99231be0a/volumes/kubernetes.io~projected/kube-api-access-dr6b4 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee53dcab1461cda6f09acdcf9dc0e732fb3f5f9340d29af4ab95cd5867f0bed0/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22b221e79df93b7ed2a9c787d47d152fce8fdd2438ac168e72ebe9eedaa29e92/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91621310a47a0b702dd2f09c18c82e07a8a8a198c6db8ae994c8ffe6bcdc4ea5/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad953721-ce75-4e69-a21c-02302526a880/volumes/kubernetes.io~projected/kube-api-access-lwcp7 DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e/volumes/kubernetes.io~projected/kube-api-access-tbj5x DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43940503-68c4-41e5-9656-4e1fe7837110/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8903afd0-dd9e-42e1-9aa0-d228f3ef611c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15953269b153210079beb81d2819bdd289e4765e7754dbfed1dc768cd90fe632/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15953269b153210 MacAddress:d2:1f:ef:21:53:2f Speed:10000 Mtu:8901} {Name:2e6d6ddb748beae MacAddress:2e:64:57:8e:80:cd Speed:10000 Mtu:8901} {Name:34f53ad17494e26 MacAddress:1a:b9:67:fb:27:eb Speed:10000 Mtu:8901} {Name:8a31d0032b9a88f MacAddress:92:88:bb:12:1a:29 Speed:10000 Mtu:8901} {Name:91621310a47a0b7 MacAddress:a6:c2:a7:4b:ad:21 Speed:10000 Mtu:8901} {Name:b1390516a228c2d MacAddress:5a:ed:af:c2:51:34 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d4:da:f5:59 Speed:0 Mtu:9001} {Name:br-int MacAddress:32:78:58:53:9c:f5 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:d4:da:f5:59 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:33:22:02:4f:e0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:09:19:01:0f:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 18:37:50.936258 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936260    4489 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 18:37:50.936511 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936314    4489 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 18:37:50.936616 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936586    4489 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 18:37:50.936737 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936617    4489 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-45.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 18:37:50.936781 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936740    4489 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 18:37:50.936781 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936748    4489 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 18:37:50.936781 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936758    4489 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:37:50.936781 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936775    4489 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:37:50.936880 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.936859    4489 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 18:37:50.937015 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937006    4489 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 18:37:50.937163 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937135    4489 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 18:37:50.937196 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937189    4489 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 18:37:50.937222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937198    4489 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 18:37:50.937222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937207    4489 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 18:37:50.937222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937216    4489 kubelet.go:405] "Adding apiserver pod source"
Nov 03 18:37:50.937333 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937223    4489 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 18:37:50.937885 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937867    4489 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 18:37:50.937885 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.937882    4489 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:37:50.938381 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.938361    4489 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 18:37:50.947087 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947072    4489 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 18:37:50.947087 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947086    4489 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:37:50.947252 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947241    4489 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 18:37:50.947434 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947424    4489 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 18:37:50.947465 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947453    4489 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 18:37:50.947557 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947547    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947561    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947569    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947574    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947580    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947585    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947590    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947595    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 18:37:50.947600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947603    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 18:37:50.947910 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947608    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 18:37:50.947910 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947624    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 18:37:50.947910 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947634    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 18:37:50.947910 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947659    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 18:37:50.947910 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.947665    4489 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 18:37:50.950895 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.950881    4489 server.go:1263] "Started kubelet"
Nov 03 18:37:50.950983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.950953    4489 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 18:37:50.951217 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.951120    4489 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 18:37:50.951297 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.951258    4489 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 18:37:50.951478 ip-10-0-143-45 systemd[1]: Started Kubernetes Kubelet.
Nov 03 18:37:50.952487 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.952268    4489 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 18:37:50.952487 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.952466    4489 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 18:37:50.954859 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.954840    4489 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 18:37:50.955184 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.955164    4489 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 18:37:50.955339 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.955267    4489 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:11 +0000 UTC" deadline="2028-06-09 01:02:17.85555529 +0000 UTC"
Nov 03 18:37:50.955339 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.955334    4489 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 18:37:50.955442 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.955347    4489 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 18:37:50.955442 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.955342    4489 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13998h24m26.900217708s"
Nov 03 18:37:50.955442 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:50.955411    4489 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:50.955585 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.955492    4489 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 18:37:50.956130 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.956114    4489 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 18:37:50.956227 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.956176    4489 factory.go:55] Registering systemd factory
Nov 03 18:37:50.956227 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.956193    4489 factory.go:223] Registration of the systemd container factory successfully
Nov 03 18:37:50.956508 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.956493    4489 factory.go:153] Registering CRI-O factory
Nov 03 18:37:50.956508 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.956509    4489 factory.go:223] Registration of the crio container factory successfully
Nov 03 18:37:50.956627 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.956550    4489 factory.go:103] Registering Raw factory
Nov 03 18:37:50.956627 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.956572    4489 manager.go:1199] Started watching for new ooms in manager
Nov 03 18:37:50.957211 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.957192    4489 manager.go:319] Starting recovery of all containers
Nov 03 18:37:50.960754 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960690    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141aa4fe-227d-45f5-a995-1b20655868fd" volumeName="kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle" seLinuxMountContext=""
Nov 03 18:37:50.960754 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960749    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43940503-68c4-41e5-9656-4e1fe7837110" volumeName="kubernetes.io/secret/43940503-68c4-41e5-9656-4e1fe7837110-agent-certs" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960766    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8903afd0-dd9e-42e1-9aa0-d228f3ef611c" volumeName="kubernetes.io/secret/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960779    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8" volumeName="kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-tuned" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960790    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8" volumeName="kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-tmp" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960798    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f" volumeName="kubernetes.io/empty-dir/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-tmp-dir" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960810    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f" volumeName="kubernetes.io/projected/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-kube-api-access-5mdp6" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960819    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8903afd0-dd9e-42e1-9aa0-d228f3ef611c" volumeName="kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-env-overrides" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960842    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="038e7f0f-0691-4006-aaee-e8b99231be0a" volumeName="kubernetes.io/empty-dir/038e7f0f-0691-4006-aaee-e8b99231be0a-tmp-dir" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960849    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141aa4fe-227d-45f5-a995-1b20655868fd" volumeName="kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960858    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141aa4fe-227d-45f5-a995-1b20655868fd" volumeName="kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960865    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e" volumeName="kubernetes.io/projected/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-kube-api-access-tbj5x" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960884    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="878a9a2c-5584-4c23-8b39-754abed5d341" volumeName="kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-multus-daemon-config" seLinuxMountContext=""
Nov 03 18:37:50.960884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960891    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8903afd0-dd9e-42e1-9aa0-d228f3ef611c" volumeName="kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960900    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93974d9c-7159-4b45-8bc7-184f9534f270" volumeName="kubernetes.io/projected/93974d9c-7159-4b45-8bc7-184f9534f270-kube-api-access-lj78x" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960908    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" volumeName="kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-binary-copy" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960916    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="344ba3f2-5bd5-4e78-bd7d-101ad41f998e" volumeName="kubernetes.io/projected/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kube-api-access-vjwzg" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960927    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="878a9a2c-5584-4c23-8b39-754abed5d341" volumeName="kubernetes.io/projected/878a9a2c-5584-4c23-8b39-754abed5d341-kube-api-access-lltc4" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960938    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8" volumeName="kubernetes.io/projected/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-kube-api-access-dsnmx" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960949    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f" volumeName="kubernetes.io/configmap/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-config-volume" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960959    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdce77a-9274-4215-9077-8281171f3c7d" volumeName="kubernetes.io/empty-dir/ffdce77a-9274-4215-9077-8281171f3c7d-data-volume" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960970    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdce77a-9274-4215-9077-8281171f3c7d" volumeName="kubernetes.io/projected/ffdce77a-9274-4215-9077-8281171f3c7d-kube-api-access-xbwqq" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960980    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e" volumeName="kubernetes.io/configmap/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-cabundle" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.960994    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" volumeName="kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961006    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141aa4fe-227d-45f5-a995-1b20655868fd" volumeName="kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961032    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="865029c1-7176-4b5c-aa12-f4a237af8e60" volumeName="kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961049    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8903afd0-dd9e-42e1-9aa0-d228f3ef611c" volumeName="kubernetes.io/projected/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-kube-api-access-jrsrc" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961062    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c167fdc3-2279-434d-bb1a-022b46d24b2c" volumeName="kubernetes.io/configmap/c167fdc3-2279-434d-bb1a-022b46d24b2c-serviceca" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961092    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" volumeName="kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961104    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="793f3faa-8090-4008-a15c-13202c13e875" volumeName="kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961118    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="878a9a2c-5584-4c23-8b39-754abed5d341" volumeName="kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-cni-binary-copy" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961129    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="038e7f0f-0691-4006-aaee-e8b99231be0a" volumeName="kubernetes.io/projected/038e7f0f-0691-4006-aaee-e8b99231be0a-kube-api-access-dr6b4" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961154    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1870fed7-afe2-43c7-ad6d-be96cf7624f2" volumeName="kubernetes.io/projected/1870fed7-afe2-43c7-ad6d-be96cf7624f2-kube-api-access-n9fsn" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961166    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e" volumeName="kubernetes.io/secret/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-key" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961176    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" volumeName="kubernetes.io/projected/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-kube-api-access-tv7wh" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961187    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad953721-ce75-4e69-a21c-02302526a880" volumeName="kubernetes.io/projected/ad953721-ce75-4e69-a21c-02302526a880-kube-api-access-lwcp7" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961200    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ec2464-111d-4061-9938-44d42bea81b7" volumeName="kubernetes.io/configmap/21ec2464-111d-4061-9938-44d42bea81b7-iptables-alerter-script" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961214    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ec2464-111d-4061-9938-44d42bea81b7" volumeName="kubernetes.io/projected/21ec2464-111d-4061-9938-44d42bea81b7-kube-api-access-4bhgv" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961226    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43940503-68c4-41e5-9656-4e1fe7837110" volumeName="kubernetes.io/configmap/43940503-68c4-41e5-9656-4e1fe7837110-konnectivity-ca" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961238    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8903afd0-dd9e-42e1-9aa0-d228f3ef611c" volumeName="kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-config" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961246    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93974d9c-7159-4b45-8bc7-184f9534f270" volumeName="kubernetes.io/secret/93974d9c-7159-4b45-8bc7-184f9534f270-cert" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961253    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c167fdc3-2279-434d-bb1a-022b46d24b2c" volumeName="kubernetes.io/projected/c167fdc3-2279-434d-bb1a-022b46d24b2c-kube-api-access-2pwsl" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961262    4489 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdce77a-9274-4215-9077-8281171f3c7d" volumeName="kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961269    4489 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 18:37:50.961400 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.961275    4489 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 18:37:50.986016 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.985960    4489 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 18:37:50.987372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.987349    4489 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 18:37:50.987372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.987373    4489 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 18:37:50.987525 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:50.987393    4489 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 18:37:50.987525 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:50.987433    4489 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 18:37:51.004820 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.004786    4489 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="41775b54ac345144cca43f96b9d468b1ebcada8ef4c0123815f70b8435042a06" exitCode=0
Nov 03 18:37:51.004931 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.004823    4489 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="6886d5d0a6f3b1d6445c44a978b008b8fed0a01db105d56561f9f485b5335d77" exitCode=0
Nov 03 18:37:51.004931 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.004835    4489 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="3734225d2fb7a07101f2e9b622fee8325099f9ffe063b6b963549720fe792544" exitCode=0
Nov 03 18:37:51.004931 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.004844    4489 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="9a6c63cf165c7db8691ffaceb5eca53f0d12f3836fea5538ef609217d07a5951" exitCode=0
Nov 03 18:37:51.004931 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.004853    4489 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="144eaebf0cfcf6641080fcc8c56a43f8f72b8e74a500a52f0939df1d38f0f585" exitCode=0
Nov 03 18:37:51.004931 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.004863    4489 generic.go:356] "Generic (PLEG): container finished" podID="3065ca9e-9bb8-4dd6-a39e-0c36efb9d866" containerID="5dee7e1af06492cedbe91b3e8fb398d534a6f716b1aeaeb5f92f5d48ba15d2c6" exitCode=0
Nov 03 18:37:51.015419 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.015394    4489 generic.go:356] "Generic (PLEG): container finished" podID="5260ce34b15d34a9c6ca404e33bcf9fb" containerID="c006ed3495eba84605737fdad66af604926e5b558c92406b9164a3b4d8ecc757" exitCode=0
Nov 03 18:37:51.026416 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.026393    4489 manager.go:324] Recovery completed
Nov 03 18:37:51.040614 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.040598    4489 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:51.041615 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.041596    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:51.041697 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.041635    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:51.041697 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.041650    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:51.043052 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043035    4489 cpu_manager.go:226] "Starting" policy="none"
Nov 03 18:37:51.043052 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043048    4489 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 18:37:51.043207 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043058    4489 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:51.043207 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043063    4489 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 18:37:51.043292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043256    4489 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 18:37:51.043292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043266    4489 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 18:37:51.043292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043280    4489 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 18:37:51.043292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043285    4489 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 18:37:51.043292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043291    4489 policy_none.go:50] "Start"
Nov 03 18:37:51.043431 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043299    4489 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 18:37:51.043431 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043307    4489 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 18:37:51.043431 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043423    4489 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 18:37:51.043431 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043430    4489 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 18:37:51.043533 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043435    4489 policy_none.go:44] "Start"
Nov 03 18:37:51.043866 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043846    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:51.043925 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043884    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:51.043925 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.043899    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:51.047688 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.047664    4489 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 18:37:51.050615 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.050570    4489 manager.go:341] "Starting Device Plugin manager"
Nov 03 18:37:51.050615 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:51.050597    4489 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 18:37:51.050615 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.050607    4489 server.go:87] "Starting device plugin registration server"
Nov 03 18:37:51.050885 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.050872    4489 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 18:37:51.050944 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.050885    4489 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 18:37:51.050990 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.050967    4489 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 18:37:51.051079 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.051054    4489 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 18:37:51.051129 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.051087    4489 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 18:37:51.051129 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.051093    4489 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 18:37:51.051225 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.051162    4489 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:37:51.051076821Z","UUID":"08af53cd-35eb-4de1-878b-073881fd111f","Handler":null,"Name":"","Endpoint":""}
Nov 03 18:37:51.052608 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:51.052553    4489 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 18:37:51.052608 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.052559    4489 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 18:37:51.052608 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.052592    4489 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 18:37:51.052608 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:51.052605    4489 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-45.us-west-2.compute.internal\" not found"
Nov 03 18:37:51.088558 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.088518    4489 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"]
Nov 03 18:37:51.088649 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.088626    4489 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:51.089433 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.089417    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:51.089477 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.089447    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:51.089477 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.089459    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:51.089693 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.089683    4489 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:51.089827 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.089814    4489 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:51.090392 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090376    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:51.090451 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090407    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:51.090451 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090418    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:51.090539 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090452    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:51.090539 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090479    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:51.090539 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090495    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:51.090678 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:51.090652    4489 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.090717 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090667    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"5260ce34b15d34a9c6ca404e33bcf9fb","Type":"ContainerStarted","Data":"8f7abd582f3ee15210d7a75c24321976f6b5a4c735e87611bcede3fa827c707d"}
Nov 03 18:37:51.090717 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090702    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"5260ce34b15d34a9c6ca404e33bcf9fb","Type":"ContainerDied","Data":"c006ed3495eba84605737fdad66af604926e5b558c92406b9164a3b4d8ecc757"}
Nov 03 18:37:51.090790 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090717    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"5260ce34b15d34a9c6ca404e33bcf9fb","Type":"ContainerStarted","Data":"081660a9101be62a24227684e0825b4772110c783a130f5bce3f90e59b39552d"}
Nov 03 18:37:51.090790 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090735    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"6e273692ac2c0053e598770db34f2c6e","Type":"ContainerStarted","Data":"48796a8eb56aab8e628ffc7430dcdc40437cca7a43d4e7196c4cb84b6266d2a8"}
Nov 03 18:37:51.090790 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090742    4489 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:51.090790 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.090747    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal" event={"ID":"6e273692ac2c0053e598770db34f2c6e","Type":"ContainerStarted","Data":"65a6bbdeaef22cb43c51670c4c18334c0d847ef00ee0b6b8df563fb63ef97fd7"}
Nov 03 18:37:51.091346 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.091330    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:51.091423 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.091357    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:51.091423 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.091367    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:51.091501 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:51.091489    4489 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-45.us-west-2.compute.internal\" not found" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.151890 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.151871    4489 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:37:51.152626 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.152613    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:37:51.152686 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.152642    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:37:51.152686 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.152652    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:37:51.152686 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.152667    4489 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.156082 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.156066    4489 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:51.156681 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.156666    4489 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:51.156772 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.156757    4489 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 18:37:51.156958 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.156944    4489 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:51.210966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.210944    4489 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:37:51.255883 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.255859    4489 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.257003 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.256979    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.257083 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.257005    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.257083 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.257023    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e273692ac2c0053e598770db34f2c6e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"6e273692ac2c0053e598770db34f2c6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.267040 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.267018    4489 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.267128 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.267079    4489 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.267565 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.267552    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeNotReady"
Nov 03 18:37:51.267636 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.267578    4489 setters.go:546] "Node became not ready" node="ip-10-0-143-45.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T18:37:51Z","lastTransitionTime":"2026-11-03T18:37:51Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 18:37:51.316481 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.316430    4489 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:37:51.316481 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:51.316469    4489 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.316481 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.316480    4489 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.327954 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.327932    4489 nodeinfomanager.go:471] existing CSINode "ip-10-0-143-45.us-west-2.compute.internal" is owned by different node (oldNodeID="a434aec4-0055-4557-bf9f-ae35810a4aef", newNodeID=""), cleaning up...
Nov 03 18:37:51.357848 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.357829    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e273692ac2c0053e598770db34f2c6e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"6e273692ac2c0053e598770db34f2c6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.357944 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.357892    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.357944 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.357854    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.358019 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.357941    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e273692ac2c0053e598770db34f2c6e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"6e273692ac2c0053e598770db34f2c6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.358019 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.357951    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.358019 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.357985    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5260ce34b15d34a9c6ca404e33bcf9fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" (UID: \"5260ce34b15d34a9c6ca404e33bcf9fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.378302 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.378285    4489 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:37:51.378356 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:51.378320    4489 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:51.938210 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:51.938180    4489 apiserver.go:52] "Watching apiserver"
Nov 03 18:37:52.028160 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.028124    4489 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:52.028292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.028282    4489 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:52.046525 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.046504    4489 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 18:37:52.046903 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.046886    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-vncch","kube-system/global-pull-secret-syncer-xr6hd","openshift-multus/multus-additional-cni-plugins-lx772","openshift-multus/multus-psxdh","openshift-ovn-kubernetes/ovnkube-node-rxtfb","openshift-image-registry/node-ca-z7fg9","openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg","openshift-service-ca/service-ca-5fffc97bdf-69t5n","openshift-cluster-node-tuning-operator/tuned-chzws","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx","openshift-insights/insights-runtime-extractor-n9565","kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-9pxqg","openshift-multus/network-metrics-daemon-7sqm4","openshift-dns/dns-default-khjmx","openshift-insights/periodic-gathering-4tm2v-tn8v7","openshift-network-operator/iptables-alerter-x54sg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal","kube-system/konnectivity-agent-b2gtz","openshift-network-diagnostics/network-check-target-m76pn"]
Nov 03 18:37:52.052261 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.052224    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:52.053974 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.053955    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.054333 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.054317    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.056595 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.056578    4489 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 18:37:52.062876 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.062859    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.062938 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.062887    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-kubernetes\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.062938 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.062902    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-conf\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.062938 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.062918    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbwqq\" (UniqueName: \"kubernetes.io/projected/ffdce77a-9274-4215-9077-8281171f3c7d-kube-api-access-xbwqq\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.063065 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.062943    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-var-lib-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063065 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.062999    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-log-socket\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063065 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063030    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwzg\" (UniqueName: \"kubernetes.io/projected/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kube-api-access-vjwzg\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.063303 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063078    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-k8s-cni-cncf-io\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.063303 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063236    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-modprobe-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.063303 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063254    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-run\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.063303 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063271    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43940503-68c4-41e5-9656-4e1fe7837110-agent-certs\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:52.063414 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063332    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-node-log\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063414 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063348    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsrc\" (UniqueName: \"kubernetes.io/projected/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-kube-api-access-jrsrc\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063414 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063370    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21ec2464-111d-4061-9938-44d42bea81b7-host-slash\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:52.063497 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063420    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwcp7\" (UniqueName: \"kubernetes.io/projected/ad953721-ce75-4e69-a21c-02302526a880-kube-api-access-lwcp7\") pod \"migrator-66df5d496b-78tkg\" (UID: \"ad953721-ce75-4e69-a21c-02302526a880\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"
Nov 03 18:37:52.063497 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063448    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-multus-certs\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.063561 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063499    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-system-cni-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.063561 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063517    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-ovn\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063561 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063534    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovn-node-metrics-cert\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063561 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063549    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-socket-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.063727 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063562    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-sys\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.063727 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063624    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.063727 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063666    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.063814 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063734    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-systemd-units\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063814 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063793    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6b4\" (UniqueName: \"kubernetes.io/projected/038e7f0f-0691-4006-aaee-e8b99231be0a-kube-api-access-dr6b4\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:52.063868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063821    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:52.063900 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063871    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdce77a-9274-4215-9077-8281171f3c7d-data-volume\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.063929 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063903    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.063957 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063931    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-env-overrides\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.063957 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063933    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffdce77a-9274-4215-9077-8281171f3c7d-data-volume\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.064011 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.063987    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-script-lib\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.064040 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064017    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-tuned\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.064070 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064041    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.064101 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064065    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cnibin\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.064101 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064088    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.064180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064112    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-tuned\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.064180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064113    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pwsl\" (UniqueName: \"kubernetes.io/projected/c167fdc3-2279-434d-bb1a-022b46d24b2c-kube-api-access-2pwsl\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:52.064238 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064178    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdh6\" (UniqueName: \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.064238 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064232    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.064300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064250    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdp6\" (UniqueName: \"kubernetes.io/projected/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-kube-api-access-5mdp6\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.064300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064267    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21ec2464-111d-4061-9938-44d42bea81b7-iptables-alerter-script\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:52.064300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064282    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnmx\" (UniqueName: \"kubernetes.io/projected/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-kube-api-access-dsnmx\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.064300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064297    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdce77a-9274-4215-9077-8281171f3c7d-crio-socket\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.064500 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064312    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-bin\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.064500 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064357    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-netd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.064500 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064389    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.064500 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064431    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038e7f0f-0691-4006-aaee-e8b99231be0a-tmp-dir\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:52.064500 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064454    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:52.064500 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064490    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhgv\" (UniqueName: \"kubernetes.io/projected/21ec2464-111d-4061-9938-44d42bea81b7-kube-api-access-4bhgv\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:52.064660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064519    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038e7f0f-0691-4006-aaee-e8b99231be0a-tmp-dir\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:52.064660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064568    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-tmp-dir\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.064660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064597    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-registration-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.064736 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064686    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.064736 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064709    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-tmp-dir\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.064736 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064716    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lltc4\" (UniqueName: \"kubernetes.io/projected/878a9a2c-5584-4c23-8b39-754abed5d341-kube-api-access-lltc4\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.064819 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064741    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c167fdc3-2279-434d-bb1a-022b46d24b2c-host\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:52.064819 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064761    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-cnibin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.064819 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064775    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-netns\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.064819 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064792    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.064819 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064810    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.064948 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064843    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-system-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.064948 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064881    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-kubelet\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.064948 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064912    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.065025 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064960    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbj5x\" (UniqueName: \"kubernetes.io/projected/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-kube-api-access-tbj5x\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:52.065025 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064978    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.065025 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.064993    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-slash\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.065025 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065009    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c167fdc3-2279-434d-bb1a-022b46d24b2c-serviceca\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:52.065025 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065024    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-device-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.065178 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065135    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-cni-binary-copy\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065215 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065176    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-bin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065215 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065194    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-conf-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065215 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065210    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-kubelet-config\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.065292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065223    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-dbus\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.065292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065237    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.065292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065268    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-lib-modules\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.065292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065289    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.065397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065305    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-socket-dir-parent\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065338    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-multus-daemon-config\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065358    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-etc-kubernetes\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065363    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.065397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065393    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-var-lib-kubelet\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.065560 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065419    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-host\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.065560 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065448    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-tmp\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.065560 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065480    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-key\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:52.065560 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065507    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-tmp\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.065560 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065516    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.065684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065574    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-systemd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.065684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065594    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.065684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065618    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-config\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.065684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065648    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-hostroot\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065672    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-cabundle\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:52.065826 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065699    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-kubelet\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.065826 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065733    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-netns\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.065826 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065771    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-etc-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.065826 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065796    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/038e7f0f-0691-4006-aaee-e8b99231be0a-hosts-file\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:52.065826 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065814    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9fsn\" (UniqueName: \"kubernetes.io/projected/1870fed7-afe2-43c7-ad6d-be96cf7624f2-kube-api-access-n9fsn\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:52.065954 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065846    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-sys-fs\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.065954 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065925    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-multus\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.065954 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065949    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93974d9c-7159-4b45-8bc7-184f9534f270-cert\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:52.066031 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.065977    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj78x\" (UniqueName: \"kubernetes.io/projected/93974d9c-7159-4b45-8bc7-184f9534f270-kube-api-access-lj78x\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:52.066059 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.066037    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysconfig\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.066086 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.066063    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-systemd\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.066194 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.066179    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-config-volume\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.066234 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.066212    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7wh\" (UniqueName: \"kubernetes.io/projected/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-kube-api-access-tv7wh\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.066262 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.066233    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43940503-68c4-41e5-9656-4e1fe7837110-konnectivity-ca\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:52.066290 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.066259    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.090431 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.090415    4489 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:37:52.090494 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.090454    4489 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:52.093437 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.093424    4489 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:37:52.093479 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.093457    4489 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-45.us-west-2.compute.internal"
Nov 03 18:37:52.104710 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.104688    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.105732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.105707    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.105822 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.105777    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.105924 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.105904    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.105997 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.105922    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.105997 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.105947    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.105997 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.105953    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 18:37:52.106167 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106130    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.106238 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106166    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 18:37:52.106238 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106219    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 18:37:52.106331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106261    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.106331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106292    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.106539 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106526    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.106624 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106609    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106643    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lzn26\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106656    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxjph\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106658    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106680    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d4mhq\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106681    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fft52\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106695    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106683    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 18:37:52.106711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106709    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106713    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4grw\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106729    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkfh8\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106742    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106748    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106752    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cxqpk\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106795    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-btttq\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106826    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.107095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.106862    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.114126 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.114109    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-env-overrides\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.114126 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.114122    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovn-node-metrics-cert\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.115386 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.115370    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c167fdc3-2279-434d-bb1a-022b46d24b2c-serviceca\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:52.115451 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.115398    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-cni-binary-copy\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.115451 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.115424    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/793f3faa-8090-4008-a15c-13202c13e875-original-pull-secret\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.115451 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.115424    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-binary-copy\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.115601 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.115461    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21ec2464-111d-4061-9938-44d42bea81b7-iptables-alerter-script\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:52.115601 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.115541    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/878a9a2c-5584-4c23-8b39-754abed5d341-multus-daemon-config\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.116504 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.116489    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-config\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.152831 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.152808    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwzg\" (UniqueName: \"kubernetes.io/projected/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kube-api-access-vjwzg\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.153413 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.153364    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7wh\" (UniqueName: \"kubernetes.io/projected/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-kube-api-access-tv7wh\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.153569 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.153542    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnmx\" (UniqueName: \"kubernetes.io/projected/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-kube-api-access-dsnmx\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.153797 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.153779    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhgv\" (UniqueName: \"kubernetes.io/projected/21ec2464-111d-4061-9938-44d42bea81b7-kube-api-access-4bhgv\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:52.153937 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.153920    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lhj\" (UniqueName: \"kubernetes.io/projected/865029c1-7176-4b5c-aa12-f4a237af8e60-kube-api-access-b5lhj\") pod \"network-check-target-m76pn\" (UID: \"865029c1-7176-4b5c-aa12-f4a237af8e60\") " pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:52.154477 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.154462    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsrc\" (UniqueName: \"kubernetes.io/projected/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-kube-api-access-jrsrc\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.157905 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.157888    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9fsn\" (UniqueName: \"kubernetes.io/projected/1870fed7-afe2-43c7-ad6d-be96cf7624f2-kube-api-access-n9fsn\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:52.158382 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.158365    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lltc4\" (UniqueName: \"kubernetes.io/projected/878a9a2c-5584-4c23-8b39-754abed5d341-kube-api-access-lltc4\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167004 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.166987    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-hostroot\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167079 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167010    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-kubelet\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167079 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167026    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-netns\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167079 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167040    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-etc-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167083    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-etc-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167092    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-kubelet\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167114    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/038e7f0f-0691-4006-aaee-e8b99231be0a-hosts-file\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:52.167212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167097    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-hostroot\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167097    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-netns\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167158    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-sys-fs\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.167212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167203    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-sys-fs\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167214    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-multus\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167178    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/038e7f0f-0691-4006-aaee-e8b99231be0a-hosts-file\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167248    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-multus\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167276    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysconfig\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167292    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-systemd\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167308    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-kubernetes\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167323    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-conf\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167327    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysconfig\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167338    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-var-lib-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167353    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-log-socket\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167356    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-kubernetes\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167369    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-k8s-cni-cncf-io\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167384    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-modprobe-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167394    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-systemd\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167401    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-run\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167404    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-var-lib-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167426    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-node-log\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167441    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-k8s-cni-cncf-io\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167453    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21ec2464-111d-4061-9938-44d42bea81b7-host-slash\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167426    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-log-socket\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167442    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-conf\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167467    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-run\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167479    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21ec2464-111d-4061-9938-44d42bea81b7-host-slash\") pod \"iptables-alerter-x54sg\" (UID: \"21ec2464-111d-4061-9938-44d42bea81b7\") " pod="openshift-network-operator/iptables-alerter-x54sg"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167483    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-multus-certs\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167499    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-modprobe-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167467    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-node-log\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167508    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-system-cni-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167526    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-system-cni-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167511    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-multus-certs\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167549    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-ovn\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167566    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-socket-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167580    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-sys\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167595    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167632    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-socket-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167655    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-sys\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167658    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-ovn\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167660    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-systemd-units\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167688    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-systemd-units\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167691    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167729    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cnibin\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167746    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167761    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167776    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cnibin\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167778    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdce77a-9274-4215-9077-8281171f3c7d-crio-socket\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167805    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffdce77a-9274-4215-9077-8281171f3c7d-crio-socket\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167812    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-bin\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167837    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-netd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167844    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-bin\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167838    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-openvswitch\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167865    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.167854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167868    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-cni-netd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167895    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-registration-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167910    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167924    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c167fdc3-2279-434d-bb1a-022b46d24b2c-host\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167966    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-cnibin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167977    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-registration-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167986    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167987    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-netns\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168010    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-run-netns\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168019    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168029    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-cnibin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168035    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-system-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.167977    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168019    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c167fdc3-2279-434d-bb1a-022b46d24b2c-host\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168051    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-kubelet\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168076    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-kubelet\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168081    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168049    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168089    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-system-cni-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168099    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-slash\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168115    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-device-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168126    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168136    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-bin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168186    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-conf-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168200    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-device-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168204    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-slash\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168190    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-host-var-lib-cni-bin\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168214    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-kubelet-config\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168240    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-dbus\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168220    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-conf-dir\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168249    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-kubelet-config\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168254    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168269    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-lib-modules\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168279    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/793f3faa-8090-4008-a15c-13202c13e875-dbus\") pod \"global-pull-secret-syncer-xr6hd\" (UID: \"793f3faa-8090-4008-a15c-13202c13e875\") " pod="kube-system/global-pull-secret-syncer-xr6hd"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168291    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-socket-dir-parent\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168313    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-etc-kubernetes\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168335    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-var-lib-kubelet\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168347    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-lib-modules\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168353    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-host\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168359    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-multus-socket-dir-parent\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168365    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/878a9a2c-5584-4c23-8b39-754abed5d341-etc-kubernetes\") pod \"multus-psxdh\" (UID: \"878a9a2c-5584-4c23-8b39-754abed5d341\") " pod="openshift-multus/multus-psxdh"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168377    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-etc-sysctl-d\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168380    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168401    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-host\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168403    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8-var-lib-kubelet\") pod \"tuned-chzws\" (UID: \"aaf6e514-f74c-4e4d-b8e9-1180fc6a7ec8\") " pod="openshift-cluster-node-tuning-operator/tuned-chzws"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168413    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-systemd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168423    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/344ba3f2-5bd5-4e78-bd7d-101ad41f998e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9v9wx\" (UID: \"344ba3f2-5bd5-4e78-bd7d-101ad41f998e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9v9wx"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168451    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-run-systemd\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168450    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.168836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.168482    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.205271 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205221    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.205271 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205224    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205293    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205296    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205327    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205327    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205357    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205359    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205361    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 18:37:52.205420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205384    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.205728 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205680    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 18:37:52.205728 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205685    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 18:37:52.205728 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205724    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.205901 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205743    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 18:37:52.205901 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205755    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.205901 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205762    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.205901 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205774    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 18:37:52.205901 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205788    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 18:37:52.206103 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.205938    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-cabundle\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:52.206176 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206114    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 18:37:52.206227 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206191    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 18:37:52.206299 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206287    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 18:37:52.206371 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206341    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zsb8x\""
Nov 03 18:37:52.206371 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206349    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 18:37:52.206371 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206356    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-config-volume\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.206511 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206391    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc7sj\""
Nov 03 18:37:52.206511 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206416    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 18:37:52.206511 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206453    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206763    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43940503-68c4-41e5-9656-4e1fe7837110-konnectivity-ca\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.206982    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bgcqc\""
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.207733    4489 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.207830    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.707802538 +0000 UTC m=+1.856497102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.207860    4489 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.207900    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.707890212 +0000 UTC m=+1.856584775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : secret "metrics-daemon-secret" not found
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.207927    4489 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:52.209461 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.207960    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:52.707950832 +0000 UTC m=+1.856645396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:52.210175 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.210063    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbwqq\" (UniqueName: \"kubernetes.io/projected/ffdce77a-9274-4215-9077-8281171f3c7d-kube-api-access-xbwqq\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.210175 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.210119    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-442ps\""
Nov 03 18:37:52.210175 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.210128    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r88kn\""
Nov 03 18:37:52.210175 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.210126    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 18:37:52.210175 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.210160    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczj\""
Nov 03 18:37:52.210473 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.210450    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6b4\" (UniqueName: \"kubernetes.io/projected/038e7f0f-0691-4006-aaee-e8b99231be0a-kube-api-access-dr6b4\") pod \"node-resolver-vncch\" (UID: \"038e7f0f-0691-4006-aaee-e8b99231be0a\") " pod="openshift-dns/node-resolver-vncch"
Nov 03 18:37:52.212780 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.212762    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zjfk4\""
Nov 03 18:37:52.212843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.212810    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dvl9\""
Nov 03 18:37:52.212927 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.212912    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 18:37:52.212980 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.212937    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2q9jp\""
Nov 03 18:37:52.212980 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.212947    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88hbk\""
Nov 03 18:37:52.213060 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.213047    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 18:37:52.213093 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.213072    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pwsl\" (UniqueName: \"kubernetes.io/projected/c167fdc3-2279-434d-bb1a-022b46d24b2c-kube-api-access-2pwsl\") pod \"node-ca-z7fg9\" (UID: \"c167fdc3-2279-434d-bb1a-022b46d24b2c\") " pod="openshift-image-registry/node-ca-z7fg9"
Nov 03 18:37:52.213292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.213277    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.213491 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.213477    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdh6\" (UniqueName: \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\") pod \"periodic-gathering-4tm2v-tn8v7\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") " pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:37:52.213636 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.213622    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43940503-68c4-41e5-9656-4e1fe7837110-agent-certs\") pod \"konnectivity-agent-b2gtz\" (UID: \"43940503-68c4-41e5-9656-4e1fe7837110\") " pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:37:52.213843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.213823    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdp6\" (UniqueName: \"kubernetes.io/projected/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-kube-api-access-5mdp6\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.213960 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.213946    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.214060 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.214047    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3065ca9e-9bb8-4dd6-a39e-0c36efb9d866-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lx772\" (UID: \"3065ca9e-9bb8-4dd6-a39e-0c36efb9d866\") " pod="openshift-multus/multus-additional-cni-plugins-lx772"
Nov 03 18:37:52.214240 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.214222    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8903afd0-dd9e-42e1-9aa0-d228f3ef611c-ovnkube-script-lib\") pod \"ovnkube-node-rxtfb\" (UID: \"8903afd0-dd9e-42e1-9aa0-d228f3ef611c\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:52.214335 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.214316    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffdce77a-9274-4215-9077-8281171f3c7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.214383 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.214338    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwcp7\" (UniqueName: \"kubernetes.io/projected/ad953721-ce75-4e69-a21c-02302526a880-kube-api-access-lwcp7\") pod \"migrator-66df5d496b-78tkg\" (UID: \"ad953721-ce75-4e69-a21c-02302526a880\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-78tkg"
Nov 03 18:37:52.214556 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.214543    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbj5x\" (UniqueName: \"kubernetes.io/projected/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-kube-api-access-tbj5x\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:52.215749 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.215732    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e-signing-key\") pod \"service-ca-5fffc97bdf-69t5n\" (UID: \"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n"
Nov 03 18:37:52.215813 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.215799    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj78x\" (UniqueName: \"kubernetes.io/projected/93974d9c-7159-4b45-8bc7-184f9534f270-kube-api-access-lj78x\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:52.216880 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.216867    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93974d9c-7159-4b45-8bc7-184f9534f270-cert\") pod \"ingress-canary-9pxqg\" (UID: \"93974d9c-7159-4b45-8bc7-184f9534f270\") " pod="openshift-ingress-canary/ingress-canary-9pxqg"
Nov 03 18:37:52.356483 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.356464    4489 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:37:52.773523 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.773490    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:52.773523 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.773524    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:52.773722 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:52.773551    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:52.773722 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.773634    4489 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:52.773722 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.773642    4489 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:52.773722 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.773649    4489 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 18:37:52.773722 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.773692    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.773679677 +0000 UTC m=+2.922374225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:52.773722 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.773703    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.773697993 +0000 UTC m=+2.922392542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:52.773722 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:52.773720    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:53.773707882 +0000 UTC m=+2.922402430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : secret "metrics-daemon-secret" not found
Nov 03 18:37:53.781223 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:53.781177    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:53.781229    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:53.781266    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:53.781342    4489 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:53.781387    4489 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:53.781418    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.781397728 +0000 UTC m=+4.930092295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:53.781439    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.781429448 +0000 UTC m=+4.930123997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:53.781469    4489 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 18:37:53.781674 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:53.781519    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:55.781503263 +0000 UTC m=+4.930197827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : secret "metrics-daemon-secret" not found
Nov 03 18:37:54.034249 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:54.034173    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7" event={"ID":"141aa4fe-227d-45f5-a995-1b20655868fd","Type":"ContainerStarted","Data":"9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192"}
Nov 03 18:37:55.796233 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:55.796201    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:55.796233 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:55.796237    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:55.796708 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:55.796256    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:55.796708 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:55.796355    4489 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:55.796708 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:55.796356    4489 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:55.796708 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:55.796356    4489 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 18:37:55.796708 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:55.796413    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.796401262 +0000 UTC m=+8.945095811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:55.796708 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:55.796431    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.796418849 +0000 UTC m=+8.945113397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : secret "metrics-daemon-secret" not found
Nov 03 18:37:55.796708 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:55.796443    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls podName:ffdce77a-9274-4215-9077-8281171f3c7d nodeName:}" failed. No retries permitted until 2026-11-03 18:37:59.796437534 +0000 UTC m=+8.945132082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n9565" (UID: "ffdce77a-9274-4215-9077-8281171f3c7d") : secret "insights-runtime-extractor-tls" not found
Nov 03 18:37:55.904705 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:55.904673    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:37:56.014530 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:56.014508    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:37:56.040297 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:56.040263    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-69t5n" event={"ID":"1fce30d6-7869-4ce9-ab2a-7c100a2c4f0e","Type":"ContainerStarted","Data":"1dbc5c75ae83f1f5ee65a80b7b9ab87a27e2c299ac2300688e2edd6b345c7269"}
Nov 03 18:37:57.299976 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:57.299947    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vncch_038e7f0f-0691-4006-aaee-e8b99231be0a/dns-node-resolver/0.log"
Nov 03 18:37:58.299905 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:58.299879    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z7fg9_c167fdc3-2279-434d-bb1a-022b46d24b2c/node-ca/0.log"
Nov 03 18:37:58.699938 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:58.699910    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9pxqg_93974d9c-7159-4b45-8bc7-184f9534f270/serve-healthcheck-canary/0.log"
Nov 03 18:37:59.101060 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:59.100982    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-78tkg_ad953721-ce75-4e69-a21c-02302526a880/migrator/0.log"
Nov 03 18:37:59.822271 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:59.822237    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:59.822271 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:59.822269    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:37:59.822712 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:59.822291    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:37:59.822712 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:59.822379    4489 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 18:37:59.822712 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:59.822383    4489 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 18:37:59.822712 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:59.822430    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls podName:ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.822417532 +0000 UTC m=+16.971112093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls") pod "dns-default-khjmx" (UID: "ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f") : secret "dns-default-metrics-tls" not found
Nov 03 18:37:59.822712 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:37:59.822442    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs podName:1870fed7-afe2-43c7-ad6d-be96cf7624f2 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:07.822436162 +0000 UTC m=+16.971130711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs") pod "network-metrics-daemon-7sqm4" (UID: "1870fed7-afe2-43c7-ad6d-be96cf7624f2") : secret "metrics-daemon-secret" not found
Nov 03 18:37:59.822712 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:59.822636    4489 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 18:37:59.824754 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:59.824734    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffdce77a-9274-4215-9077-8281171f3c7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n9565\" (UID: \"ffdce77a-9274-4215-9077-8281171f3c7d\") " pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:37:59.854808 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:37:59.854786    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n9565"
Nov 03 18:38:00.065337 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:00.065311    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b2gtz"
Nov 03 18:38:00.218509 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:00.218478    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdce77a_9274_4215_9077_8281171f3c7d.slice/crio-a0748c351a811f5957bf5198c4c39f4410d82ecabd9e489dc2353fc001078329 WatchSource:0}: Error finding container a0748c351a811f5957bf5198c4c39f4410d82ecabd9e489dc2353fc001078329: Status 404 returned error can't find the container with id a0748c351a811f5957bf5198c4c39f4410d82ecabd9e489dc2353fc001078329
Nov 03 18:38:00.219799 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:00.219778    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n9565"]
Nov 03 18:38:01.057640 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:01.057604    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9565" event={"ID":"ffdce77a-9274-4215-9077-8281171f3c7d","Type":"ContainerStarted","Data":"a761b33842a081aed4daa1f8e16d4725398c62d49d0f944b7390be2a31b38b6a"}
Nov 03 18:38:01.058340 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:01.058303    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9565" event={"ID":"ffdce77a-9274-4215-9077-8281171f3c7d","Type":"ContainerStarted","Data":"a0748c351a811f5957bf5198c4c39f4410d82ecabd9e489dc2353fc001078329"}
Nov 03 18:38:01.255255 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:01.255219    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:38:01.255484 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:01.255468    4489 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:38:01.560564 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:01.560537    4489 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-45.us-west-2.compute.internal" event="NodeReady"
Nov 03 18:38:02.062329 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:02.062300    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9565" event={"ID":"ffdce77a-9274-4215-9077-8281171f3c7d","Type":"ContainerStarted","Data":"f4cf7428d5a8884d9e141230e4e5b9f63f1b38e2406c1fe0d2b340b002948329"}
Nov 03 18:38:04.070239 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:04.070203    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n9565" event={"ID":"ffdce77a-9274-4215-9077-8281171f3c7d","Type":"ContainerStarted","Data":"94b536dbc21a01736b940bc5e39fa54190c7b72ed17003f6e468d0b1f1fb659d"}
Nov 03 18:38:07.881939 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:07.881904    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:38:07.881939 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:07.881943    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:38:07.884464 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:07.884435    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f-metrics-tls\") pod \"dns-default-khjmx\" (UID: \"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f\") " pod="openshift-dns/dns-default-khjmx"
Nov 03 18:38:07.884582 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:07.884441    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1870fed7-afe2-43c7-ad6d-be96cf7624f2-metrics-certs\") pod \"network-metrics-daemon-7sqm4\" (UID: \"1870fed7-afe2-43c7-ad6d-be96cf7624f2\") " pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:38:07.952797 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:07.952768    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7sqm4"
Nov 03 18:38:07.954356 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:07.954334    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khjmx"
Nov 03 18:38:08.377702 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:08.377671    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad31b4da_d26a_47a1_9f84_9fdb1c5a4d2f.slice/crio-46e7aae979f49a3e00ca0fb05b1648b83d77e4b3670833d7d7b4f3dacd89dc44 WatchSource:0}: Error finding container 46e7aae979f49a3e00ca0fb05b1648b83d77e4b3670833d7d7b4f3dacd89dc44: Status 404 returned error can't find the container with id 46e7aae979f49a3e00ca0fb05b1648b83d77e4b3670833d7d7b4f3dacd89dc44
Nov 03 18:38:08.379483 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:08.379459    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khjmx"]
Nov 03 18:38:08.424344 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:08.424318    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1870fed7_afe2_43c7_ad6d_be96cf7624f2.slice/crio-e2f9aae0e8de3c88ba94c38376d72dca8589a9568776b36d1e462ea0bf0bcd68 WatchSource:0}: Error finding container e2f9aae0e8de3c88ba94c38376d72dca8589a9568776b36d1e462ea0bf0bcd68: Status 404 returned error can't find the container with id e2f9aae0e8de3c88ba94c38376d72dca8589a9568776b36d1e462ea0bf0bcd68
Nov 03 18:38:08.431287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:08.431266    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7sqm4"]
Nov 03 18:38:09.085343 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.085313    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7sqm4" event={"ID":"1870fed7-afe2-43c7-ad6d-be96cf7624f2","Type":"ContainerStarted","Data":"e2f9aae0e8de3c88ba94c38376d72dca8589a9568776b36d1e462ea0bf0bcd68"}
Nov 03 18:38:09.086465 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.086441    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khjmx" event={"ID":"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f","Type":"ContainerStarted","Data":"46e7aae979f49a3e00ca0fb05b1648b83d77e4b3670833d7d7b4f3dacd89dc44"}
Nov 03 18:38:09.470237 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.470202    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-584f9d55d8-hb897"]
Nov 03 18:38:09.472408 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.472383    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.535247 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.535209    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 18:38:09.586154 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.586105    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n2c2b\""
Nov 03 18:38:09.586373 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.586357    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 18:38:09.586852 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.586837    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 18:38:09.587286 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.587215    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-584f9d55d8-hb897"]
Nov 03 18:38:09.594308 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594282    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3e859e5c-2479-498c-9214-b4e30dbef68d-image-registry-private-configuration\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.594443 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594313    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3e859e5c-2479-498c-9214-b4e30dbef68d-registry-certificates\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.594443 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594335    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3e859e5c-2479-498c-9214-b4e30dbef68d-installation-pull-secrets\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.594443 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594365    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e859e5c-2479-498c-9214-b4e30dbef68d-trusted-ca\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.594613 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594511    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9nsb\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-kube-api-access-p9nsb\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.594613 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594558    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-registry-tls\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.594733 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594685    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3e859e5c-2479-498c-9214-b4e30dbef68d-ca-trust-extracted\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.594823 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.594803    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-bound-sa-token\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.690804 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.690764    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 18:38:09.696237 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696201    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9nsb\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-kube-api-access-p9nsb\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.696397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696245    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-registry-tls\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.696397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696272    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3e859e5c-2479-498c-9214-b4e30dbef68d-ca-trust-extracted\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.696397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696300    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-bound-sa-token\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.696397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696341    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3e859e5c-2479-498c-9214-b4e30dbef68d-image-registry-private-configuration\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.696397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696368    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3e859e5c-2479-498c-9214-b4e30dbef68d-registry-certificates\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.696397 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696391    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3e859e5c-2479-498c-9214-b4e30dbef68d-installation-pull-secrets\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.697254 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696415    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e859e5c-2479-498c-9214-b4e30dbef68d-trusted-ca\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.697254 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.696712    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3e859e5c-2479-498c-9214-b4e30dbef68d-ca-trust-extracted\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.697424 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.697397    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3e859e5c-2479-498c-9214-b4e30dbef68d-trusted-ca\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.697594 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.697575    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3e859e5c-2479-498c-9214-b4e30dbef68d-registry-certificates\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.699567 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.699542    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3e859e5c-2479-498c-9214-b4e30dbef68d-installation-pull-secrets\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.699680 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.699579    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3e859e5c-2479-498c-9214-b4e30dbef68d-image-registry-private-configuration\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.700024 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.700003    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-registry-tls\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.763067 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.762995    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-bound-sa-token\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.765401 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.765371    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9nsb\" (UniqueName: \"kubernetes.io/projected/3e859e5c-2479-498c-9214-b4e30dbef68d-kube-api-access-p9nsb\") pod \"image-registry-584f9d55d8-hb897\" (UID: \"3e859e5c-2479-498c-9214-b4e30dbef68d\") " pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:09.785896 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:09.785867    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:10.265352 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:10.265314    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e859e5c_2479_498c_9214_b4e30dbef68d.slice/crio-ee243e52cb2f843dd1ee78a628f1de949e574f385fa302783173f74af74114b6 WatchSource:0}: Error finding container ee243e52cb2f843dd1ee78a628f1de949e574f385fa302783173f74af74114b6: Status 404 returned error can't find the container with id ee243e52cb2f843dd1ee78a628f1de949e574f385fa302783173f74af74114b6
Nov 03 18:38:10.266358 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:10.266333    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-584f9d55d8-hb897"]
Nov 03 18:38:10.766325 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:10.766294    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxtfb"
Nov 03 18:38:11.093897 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.093853    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7sqm4" event={"ID":"1870fed7-afe2-43c7-ad6d-be96cf7624f2","Type":"ContainerStarted","Data":"eecc1155d28c8e750b6c7cefb624028aab7da7fb509d7b8d7c39ca7a6c08ba99"}
Nov 03 18:38:11.094098 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.093929    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7sqm4" event={"ID":"1870fed7-afe2-43c7-ad6d-be96cf7624f2","Type":"ContainerStarted","Data":"e8247087c98bda48f8b752e00306c4c33eed2f644cbd49af75c48857eb6cfb45"}
Nov 03 18:38:11.095751 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.095726    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khjmx" event={"ID":"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f","Type":"ContainerStarted","Data":"3c9edbb9a859b02a851c7044d264a7ecf10210912ce197b3098b08601b79a06b"}
Nov 03 18:38:11.095868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.095769    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-khjmx"
Nov 03 18:38:11.095868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.095790    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khjmx" event={"ID":"ad31b4da-d26a-47a1-9f84-9fdb1c5a4d2f","Type":"ContainerStarted","Data":"68302ff444d1263c2c3c97235e9fa3ebc64cb242370811b63441adad2b374e7a"}
Nov 03 18:38:11.097234 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.097212    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-584f9d55d8-hb897" event={"ID":"3e859e5c-2479-498c-9214-b4e30dbef68d","Type":"ContainerStarted","Data":"f9792ba772e139c59f2746d00ee141c5acb71c8855c1542b7ee68eb14b241d67"}
Nov 03 18:38:11.097304 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.097250    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-584f9d55d8-hb897" event={"ID":"3e859e5c-2479-498c-9214-b4e30dbef68d","Type":"ContainerStarted","Data":"ee243e52cb2f843dd1ee78a628f1de949e574f385fa302783173f74af74114b6"}
Nov 03 18:38:11.097304 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.097284    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:11.341830 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:11.341770    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-584f9d55d8-hb897" podStartSLOduration=2.341748152 podStartE2EDuration="2.341748152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:11.340463404 +0000 UTC m=+20.489157975" watchObservedRunningTime="2026-11-03 18:38:11.341748152 +0000 UTC m=+20.490442717"
Nov 03 18:38:13.329746 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.329705    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"]
Nov 03 18:38:13.339698 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.339668    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"
Nov 03 18:38:13.387331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.387308    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"]
Nov 03 18:38:13.425435 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.425402    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/91ec170f-ad66-4671-9a63-0be41bbc2f00-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-r9b4h\" (UID: \"91ec170f-ad66-4671-9a63-0be41bbc2f00\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"
Nov 03 18:38:13.438539 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.438519    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 18:38:13.438617 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.438582    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nmbv5\""
Nov 03 18:38:13.526432 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.526376    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/91ec170f-ad66-4671-9a63-0be41bbc2f00-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-r9b4h\" (UID: \"91ec170f-ad66-4671-9a63-0be41bbc2f00\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"
Nov 03 18:38:13.528802 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.528783    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/91ec170f-ad66-4671-9a63-0be41bbc2f00-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-r9b4h\" (UID: \"91ec170f-ad66-4671-9a63-0be41bbc2f00\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"
Nov 03 18:38:13.655994 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:13.655962    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"
Nov 03 18:38:14.017901 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:14.017868    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ec170f_ad66_4671_9a63_0be41bbc2f00.slice/crio-fc410434c4e349948d7937481a38446ae7c08de95cd23c225d573fef12d722fe WatchSource:0}: Error finding container fc410434c4e349948d7937481a38446ae7c08de95cd23c225d573fef12d722fe: Status 404 returned error can't find the container with id fc410434c4e349948d7937481a38446ae7c08de95cd23c225d573fef12d722fe
Nov 03 18:38:14.019622 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:14.019574    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"]
Nov 03 18:38:14.112054 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:14.112017    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h" event={"ID":"91ec170f-ad66-4671-9a63-0be41bbc2f00","Type":"ContainerStarted","Data":"fc410434c4e349948d7937481a38446ae7c08de95cd23c225d573fef12d722fe"}
Nov 03 18:38:16.120385 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:16.120345    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h" event={"ID":"91ec170f-ad66-4671-9a63-0be41bbc2f00","Type":"ContainerStarted","Data":"8c077a95f8f83c4ac07cb1534e0c49b0330e58ebd84cbcd510cf357982a2722b"}
Nov 03 18:38:16.120872 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:16.120554    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"
Nov 03 18:38:16.125372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:16.125351    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h"
Nov 03 18:38:16.364567 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:16.364515    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r9b4h" podStartSLOduration=1.893325017 podStartE2EDuration="3.364503735s" totalImagesPullingTime="1.471178718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:16.24384703 +0000 UTC m=+25.392541602" watchObservedRunningTime="2026-11-03 18:38:16.364503735 +0000 UTC m=+25.513198305"
Nov 03 18:38:17.528757 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:38:17.528723    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:38:21.106067 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:21.106040    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-khjmx"
Nov 03 18:38:25.521220 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.521192    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w9jfb"]
Nov 03 18:38:25.524183 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.524166    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603336 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603291    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrbn\" (UniqueName: \"kubernetes.io/projected/c6efa29b-d526-4d3c-b23c-b21435f0c72c-kube-api-access-xbrbn\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603336 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603337    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-textfile\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603536 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603355    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-tls\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603536 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603470    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-accelerators-collector-config\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603536 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603500    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-root\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603536 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603516    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603536 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603532    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-sys\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603726 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603612    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6efa29b-d526-4d3c-b23c-b21435f0c72c-metrics-client-ca\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.603726 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.603683    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-wtmp\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.633474 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.633444    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 18:38:25.633474 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.633456    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 18:38:25.633734 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.633481    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 18:38:25.633734 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.633536    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nx4fl\""
Nov 03 18:38:25.633734 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.633560    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 18:38:25.633734 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.633612    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 18:38:25.633734 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.633659    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 18:38:25.704896 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.704867    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6efa29b-d526-4d3c-b23c-b21435f0c72c-metrics-client-ca\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705054 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.704913    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-wtmp\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705054 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705022    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-wtmp\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705054 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705043    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrbn\" (UniqueName: \"kubernetes.io/projected/c6efa29b-d526-4d3c-b23c-b21435f0c72c-kube-api-access-xbrbn\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705255 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705086    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-textfile\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705255 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705130    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-tls\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705255 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705174    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-accelerators-collector-config\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705404 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705306    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-root\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705404 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705337    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705404 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705366    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-sys\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705528 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705414    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-root\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705528 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705449    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6efa29b-d526-4d3c-b23c-b21435f0c72c-sys\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705528 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705454    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-textfile\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705642 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705621    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6efa29b-d526-4d3c-b23c-b21435f0c72c-metrics-client-ca\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.705680 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.705657    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-accelerators-collector-config\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.707660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.707642    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.707713 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.707674    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6efa29b-d526-4d3c-b23c-b21435f0c72c-node-exporter-tls\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.772633 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.772578    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrbn\" (UniqueName: \"kubernetes.io/projected/c6efa29b-d526-4d3c-b23c-b21435f0c72c-kube-api-access-xbrbn\") pod \"node-exporter-w9jfb\" (UID: \"c6efa29b-d526-4d3c-b23c-b21435f0c72c\") " pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.836287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:25.836250    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w9jfb"
Nov 03 18:38:25.847096 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:25.847070    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6efa29b_d526_4d3c_b23c_b21435f0c72c.slice/crio-d07c9466bcc980185caa1e20ac6eb8d84f4c81d80bf908189e8495ca2a2a17e6 WatchSource:0}: Error finding container d07c9466bcc980185caa1e20ac6eb8d84f4c81d80bf908189e8495ca2a2a17e6: Status 404 returned error can't find the container with id d07c9466bcc980185caa1e20ac6eb8d84f4c81d80bf908189e8495ca2a2a17e6
Nov 03 18:38:26.017084 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:26.017058    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m76pn"
Nov 03 18:38:26.151724 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:26.151654    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9jfb" event={"ID":"c6efa29b-d526-4d3c-b23c-b21435f0c72c","Type":"ContainerStarted","Data":"d07c9466bcc980185caa1e20ac6eb8d84f4c81d80bf908189e8495ca2a2a17e6"}
Nov 03 18:38:28.158618 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:28.158581    4489 generic.go:356] "Generic (PLEG): container finished" podID="c6efa29b-d526-4d3c-b23c-b21435f0c72c" containerID="f712954a123137b3c179450def97cd98ff85f924b1d035a265d5dd58a1412557" exitCode=0
Nov 03 18:38:28.158618 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:28.158618    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9jfb" event={"ID":"c6efa29b-d526-4d3c-b23c-b21435f0c72c","Type":"ContainerDied","Data":"f712954a123137b3c179450def97cd98ff85f924b1d035a265d5dd58a1412557"}
Nov 03 18:38:29.163809 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:29.163768    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9jfb" event={"ID":"c6efa29b-d526-4d3c-b23c-b21435f0c72c","Type":"ContainerStarted","Data":"cadb93b1a7b5f250ece088ded18accf193f08e9b2ab8d17f6795293831cd98ab"}
Nov 03 18:38:29.164241 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:29.163819    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9jfb" event={"ID":"c6efa29b-d526-4d3c-b23c-b21435f0c72c","Type":"ContainerStarted","Data":"bb9169fbc95db6448fb2cb48b0289d152f454cc72069132ab86804b31069752b"}
Nov 03 18:38:29.339973 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:29.339919    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w9jfb" podStartSLOduration=2.738382153 podStartE2EDuration="4.339903034s" totalImagesPullingTime="1.527277338s" totalInitContainerRuntime="74.243543ms" isStatefulPod=true podCreationTimestamp="2026-11-03 18:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:29.339416966 +0000 UTC m=+38.488111559" watchObservedRunningTime="2026-11-03 18:38:29.339903034 +0000 UTC m=+38.488597635"
Nov 03 18:38:29.791544 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:29.791511    4489 patch_prober.go:29] interesting pod/image-registry-584f9d55d8-hb897 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 18:38:29.791721 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:29.791561    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-584f9d55d8-hb897" podUID="3e859e5c-2479-498c-9214-b4e30dbef68d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:38:30.262554 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.262524    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"]
Nov 03 18:38:30.264324 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.264309    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.320019 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.319987    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 18:38:30.320204 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.320036    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7m77d\""
Nov 03 18:38:30.321208 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.321188    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"]
Nov 03 18:38:30.339782 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.339754    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptzk\" (UniqueName: \"kubernetes.io/projected/d4e93fd2-3183-488f-b9ba-15cfe67b19a2-kube-api-access-vptzk\") pod \"monitoring-plugin-77f4f76594-5svjz\" (UID: \"d4e93fd2-3183-488f-b9ba-15cfe67b19a2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.339896 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.339804    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4e93fd2-3183-488f-b9ba-15cfe67b19a2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5svjz\" (UID: \"d4e93fd2-3183-488f-b9ba-15cfe67b19a2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.440568 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.440538    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptzk\" (UniqueName: \"kubernetes.io/projected/d4e93fd2-3183-488f-b9ba-15cfe67b19a2-kube-api-access-vptzk\") pod \"monitoring-plugin-77f4f76594-5svjz\" (UID: \"d4e93fd2-3183-488f-b9ba-15cfe67b19a2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.440734 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.440597    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4e93fd2-3183-488f-b9ba-15cfe67b19a2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5svjz\" (UID: \"d4e93fd2-3183-488f-b9ba-15cfe67b19a2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.443125 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.443105    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4e93fd2-3183-488f-b9ba-15cfe67b19a2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5svjz\" (UID: \"d4e93fd2-3183-488f-b9ba-15cfe67b19a2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.503724 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.503688    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptzk\" (UniqueName: \"kubernetes.io/projected/d4e93fd2-3183-488f-b9ba-15cfe67b19a2-kube-api-access-vptzk\") pod \"monitoring-plugin-77f4f76594-5svjz\" (UID: \"d4e93fd2-3183-488f-b9ba-15cfe67b19a2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.574344 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.574260    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:30.937767 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:30.937740    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e93fd2_3183_488f_b9ba_15cfe67b19a2.slice/crio-51cfa4d47cb018ce410d360216ee3ec6deb11d682e0272d0fcb4996d071a168d WatchSource:0}: Error finding container 51cfa4d47cb018ce410d360216ee3ec6deb11d682e0272d0fcb4996d071a168d: Status 404 returned error can't find the container with id 51cfa4d47cb018ce410d360216ee3ec6deb11d682e0272d0fcb4996d071a168d
Nov 03 18:38:30.939559 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:30.939536    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"]
Nov 03 18:38:31.171235 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:31.171160    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz" event={"ID":"d4e93fd2-3183-488f-b9ba-15cfe67b19a2","Type":"ContainerStarted","Data":"51cfa4d47cb018ce410d360216ee3ec6deb11d682e0272d0fcb4996d071a168d"}
Nov 03 18:38:32.108676 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:32.108648    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-584f9d55d8-hb897"
Nov 03 18:38:34.183306 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:34.183264    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz" event={"ID":"d4e93fd2-3183-488f-b9ba-15cfe67b19a2","Type":"ContainerStarted","Data":"b11a8e720933eb7ddaf027628de197bcd990bffcb12aece8c557da1969cc4134"}
Nov 03 18:38:34.183724 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:34.183471    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:34.188245 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:34.188224    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz"
Nov 03 18:38:34.304577 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:34.304526    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5svjz" podStartSLOduration=2.107108858 podStartE2EDuration="4.304512091s" totalImagesPullingTime="2.197403233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:34.304109655 +0000 UTC m=+43.452804222" watchObservedRunningTime="2026-11-03 18:38:34.304512091 +0000 UTC m=+43.453206656"
Nov 03 18:38:38.483272 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.483238    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77898cc49b-qqkfs"]
Nov 03 18:38:38.486061 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.486043    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.545375 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.545347    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 18:38:38.600876 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.600841    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 18:38:38.600876 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.600879    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t7k78\""
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.600930    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.600966    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.600972    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ms8f\" (UniqueName: \"kubernetes.io/projected/648a2643-630c-4a9b-aac1-47351c37805b-kube-api-access-9ms8f\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601002    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601008    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-oauth-serving-cert\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601039    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-oauth-config\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601054    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601073    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 18:38:38.601088 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601082    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-serving-cert\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.601540 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601210    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-service-ca\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.601540 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601302    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/648a2643-630c-4a9b-aac1-47351c37805b-tmp\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.601540 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601362    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-console-config\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.602018 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.601997    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77898cc49b-qqkfs"]
Nov 03 18:38:38.702724 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.702694    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-console-config\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.702724 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.702730    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ms8f\" (UniqueName: \"kubernetes.io/projected/648a2643-630c-4a9b-aac1-47351c37805b-kube-api-access-9ms8f\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.702941 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.702751    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-oauth-serving-cert\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.702941 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.702778    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-oauth-config\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.702941 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.702813    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-serving-cert\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.703086 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.702949    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-service-ca\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.703086 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.703011    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/648a2643-630c-4a9b-aac1-47351c37805b-tmp\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.703402 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.703380    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/648a2643-630c-4a9b-aac1-47351c37805b-tmp\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.703597 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.703507    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-oauth-serving-cert\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.703597 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.703515    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-console-config\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.703597 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.703610    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-service-ca\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.705382 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.705364    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-serving-cert\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.705474 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.705371    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-oauth-config\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.765981 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.765921    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ms8f\" (UniqueName: \"kubernetes.io/projected/648a2643-630c-4a9b-aac1-47351c37805b-kube-api-access-9ms8f\") pod \"console-77898cc49b-qqkfs\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") " pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:38.796706 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:38.796682    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:39.172074 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:39.172046    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648a2643_630c_4a9b_aac1_47351c37805b.slice/crio-32ff578470c1cb0169a1c24857c1dc2b6f99c9df429a62c2ced41761ea37df84 WatchSource:0}: Error finding container 32ff578470c1cb0169a1c24857c1dc2b6f99c9df429a62c2ced41761ea37df84: Status 404 returned error can't find the container with id 32ff578470c1cb0169a1c24857c1dc2b6f99c9df429a62c2ced41761ea37df84
Nov 03 18:38:39.174440 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:39.174414    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77898cc49b-qqkfs"]
Nov 03 18:38:39.199415 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:39.199393    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77898cc49b-qqkfs" event={"ID":"648a2643-630c-4a9b-aac1-47351c37805b","Type":"ContainerStarted","Data":"32ff578470c1cb0169a1c24857c1dc2b6f99c9df429a62c2ced41761ea37df84"}
Nov 03 18:38:44.215802 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:44.215768    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77898cc49b-qqkfs" event={"ID":"648a2643-630c-4a9b-aac1-47351c37805b","Type":"ContainerStarted","Data":"f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3"}
Nov 03 18:38:44.388552 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:44.388449    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77898cc49b-qqkfs" podStartSLOduration=1.426732261 podStartE2EDuration="6.388437156s" totalImagesPullingTime="4.961704895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:44.387615818 +0000 UTC m=+53.536310389" watchObservedRunningTime="2026-11-03 18:38:44.388437156 +0000 UTC m=+53.537131726"
Nov 03 18:38:46.038776 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:46.038728    4489 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b2gtz_43940503-68c4-41e5-9656-4e1fe7837110/konnectivity-agent/0.log" limit=1024
Nov 03 18:38:47.528961 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:38:47.528928    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:38:48.797058 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:48.797024    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:48.797528 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:48.797076    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:48.801925 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:48.801902    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:49.233586 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:49.233562    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:38:50.368713 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.368687    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6557849fc5-cpjm9"]
Nov 03 18:38:50.371317 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.371300    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488244 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488216    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-config\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488391 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488251    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2884fc67-ce08-4844-bbfd-ed36f3a12a61-tmp\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488391 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488267    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-trusted-ca-bundle\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488391 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488287    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-oauth-config\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488503 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488438    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-oauth-serving-cert\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488503 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488492    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ggf\" (UniqueName: \"kubernetes.io/projected/2884fc67-ce08-4844-bbfd-ed36f3a12a61-kube-api-access-t5ggf\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488566 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488532    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-service-ca\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.488600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.488588    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-serving-cert\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.582994 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.582961    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 18:38:50.584014 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.583990    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6557849fc5-cpjm9"]
Nov 03 18:38:50.589554 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589534    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2884fc67-ce08-4844-bbfd-ed36f3a12a61-tmp\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.589626 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589562    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-trusted-ca-bundle\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.589626 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589583    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-oauth-config\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.589626 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589618    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-oauth-serving-cert\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.589723 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589646    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggf\" (UniqueName: \"kubernetes.io/projected/2884fc67-ce08-4844-bbfd-ed36f3a12a61-kube-api-access-t5ggf\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.589723 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589686    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-service-ca\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.589723 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589711    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-serving-cert\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.589895 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589870    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-config\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.590004 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.589985    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2884fc67-ce08-4844-bbfd-ed36f3a12a61-tmp\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.590459 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.590436    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-service-ca\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.590593 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.590572    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-config\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.590727 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.590707    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-trusted-ca-bundle\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.590763 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.590720    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-oauth-serving-cert\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.592100 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.592073    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-oauth-config\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.592290 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.592273    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-serving-cert\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.652577 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.652517    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ggf\" (UniqueName: \"kubernetes.io/projected/2884fc67-ce08-4844-bbfd-ed36f3a12a61-kube-api-access-t5ggf\") pod \"console-6557849fc5-cpjm9\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") " pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:50.681120 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:50.681098    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:38:51.049033 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:38:51.049000    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2884fc67_ce08_4844_bbfd_ed36f3a12a61.slice/crio-1e849e3a6f5d90319c614c4d5f78c6d2b1e14c331164150597e392b00b309d81 WatchSource:0}: Error finding container 1e849e3a6f5d90319c614c4d5f78c6d2b1e14c331164150597e392b00b309d81: Status 404 returned error can't find the container with id 1e849e3a6f5d90319c614c4d5f78c6d2b1e14c331164150597e392b00b309d81
Nov 03 18:38:51.051125 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:51.051101    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6557849fc5-cpjm9"]
Nov 03 18:38:51.241724 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:51.241693    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6557849fc5-cpjm9" event={"ID":"2884fc67-ce08-4844-bbfd-ed36f3a12a61","Type":"ContainerStarted","Data":"f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5"}
Nov 03 18:38:51.241883 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:51.241738    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6557849fc5-cpjm9" event={"ID":"2884fc67-ce08-4844-bbfd-ed36f3a12a61","Type":"ContainerStarted","Data":"1e849e3a6f5d90319c614c4d5f78c6d2b1e14c331164150597e392b00b309d81"}
Nov 03 18:38:51.367572 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:38:51.367477    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6557849fc5-cpjm9" podStartSLOduration=1.36746536 podStartE2EDuration="1.36746536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:51.366672771 +0000 UTC m=+60.515367341" watchObservedRunningTime="2026-11-03 18:38:51.36746536 +0000 UTC m=+60.516159931"
Nov 03 18:39:00.682095 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:00.682046    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:39:00.682593 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:00.682116    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:39:00.686828 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:00.686811    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:39:01.276664 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:01.276635    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:39:01.430395 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:01.430367    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77898cc49b-qqkfs"]
Nov 03 18:39:17.527782 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:39:17.527744    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:39:22.372041 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.372008    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4tm2v-tn8v7"]
Nov 03 18:39:22.372497 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.372249    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7" podUID="141aa4fe-227d-45f5-a995-1b20655868fd" containerName="insights-gathering" containerID="cri-o://9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192" gracePeriod=30
Nov 03 18:39:22.636798 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.636775    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:39:22.714835 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.714813    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\" (UniqueName: \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\") pod \"141aa4fe-227d-45f5-a995-1b20655868fd\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") "
Nov 03 18:39:22.714962 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.714882    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\") pod \"141aa4fe-227d-45f5-a995-1b20655868fd\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") "
Nov 03 18:39:22.715257 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.715234    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle" pod "141aa4fe-227d-45f5-a995-1b20655868fd" (UID: "141aa4fe-227d-45f5-a995-1b20655868fd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:22.717182 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.717155    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6" pod "141aa4fe-227d-45f5-a995-1b20655868fd" (UID: "141aa4fe-227d-45f5-a995-1b20655868fd"). InnerVolumeSpecName "kube-api-access-ftdh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:22.815411 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.815385    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\") pod \"141aa4fe-227d-45f5-a995-1b20655868fd\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") "
Nov 03 18:39:22.815541 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.815421    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\") pod \"141aa4fe-227d-45f5-a995-1b20655868fd\" (UID: \"141aa4fe-227d-45f5-a995-1b20655868fd\") "
Nov 03 18:39:22.815577 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.815546    4489 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-trusted-ca-bundle\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:22.815577 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.815557    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftdh6\" (UniqueName: \"kubernetes.io/projected/141aa4fe-227d-45f5-a995-1b20655868fd-kube-api-access-ftdh6\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:22.815757 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.815734    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle" pod "141aa4fe-227d-45f5-a995-1b20655868fd" (UID: "141aa4fe-227d-45f5-a995-1b20655868fd"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:22.815791 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.815742    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path" pod "141aa4fe-227d-45f5-a995-1b20655868fd" (UID: "141aa4fe-227d-45f5-a995-1b20655868fd"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:22.916657 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.916637    4489 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141aa4fe-227d-45f5-a995-1b20655868fd-service-ca-bundle\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:22.916657 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:22.916657    4489 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/141aa4fe-227d-45f5-a995-1b20655868fd-archives-path\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:23.341319 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.341227    4489 generic.go:356] "Generic (PLEG): container finished" podID="141aa4fe-227d-45f5-a995-1b20655868fd" containerID="9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192" exitCode=2
Nov 03 18:39:23.341319 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.341277    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7"
Nov 03 18:39:23.341319 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.341277    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7" event={"ID":"141aa4fe-227d-45f5-a995-1b20655868fd","Type":"ContainerDied","Data":"9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192"}
Nov 03 18:39:23.341611 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.341325    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4tm2v-tn8v7" event={"ID":"141aa4fe-227d-45f5-a995-1b20655868fd","Type":"ContainerDied","Data":"b1390516a228c2d0cf6c365b36a69e2d953ba48a5c020561f243569e41e5b4b1"}
Nov 03 18:39:23.341611 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.341366    4489 scope.go:122] "RemoveContainer" containerID="9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192"
Nov 03 18:39:23.349026 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.349007    4489 scope.go:122] "RemoveContainer" containerID="9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192"
Nov 03 18:39:23.349302 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:39:23.349281    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192\": container with ID starting with 9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192 not found: ID does not exist" containerID="9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192"
Nov 03 18:39:23.349372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.349311    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192"} err="failed to get container status \"9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192\": rpc error: code = NotFound desc = could not find container \"9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192\": container with ID starting with 9a946aeb014bc031ff9daf7e21f5acfa8835891025f6f14bb9e4081a05e52192 not found: ID does not exist"
Nov 03 18:39:23.513374 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.513341    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4tm2v-tn8v7"]
Nov 03 18:39:23.565239 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:23.565215    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4tm2v-tn8v7"]
Nov 03 18:39:24.991842 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:24.991809    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141aa4fe-227d-45f5-a995-1b20655868fd" path="/var/lib/kubelet/pods/141aa4fe-227d-45f5-a995-1b20655868fd/volumes"
Nov 03 18:39:26.451410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.451358    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-77898cc49b-qqkfs" podUID="648a2643-630c-4a9b-aac1-47351c37805b" containerName="console" containerID="cri-o://f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3" gracePeriod=15
Nov 03 18:39:26.697025 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.697001    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:39:26.744179 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744117    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648a2643-630c-4a9b-aac1-47351c37805b-kube-api-access-9ms8f\" (UniqueName: \"kubernetes.io/projected/648a2643-630c-4a9b-aac1-47351c37805b-kube-api-access-9ms8f\") pod \"648a2643-630c-4a9b-aac1-47351c37805b\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") "
Nov 03 18:39:26.744276 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744183    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-console-config\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-console-config\") pod \"648a2643-630c-4a9b-aac1-47351c37805b\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") "
Nov 03 18:39:26.744276 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744235    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648a2643-630c-4a9b-aac1-47351c37805b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/648a2643-630c-4a9b-aac1-47351c37805b-tmp\") pod \"648a2643-630c-4a9b-aac1-47351c37805b\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") "
Nov 03 18:39:26.744342 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744274    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-oauth-serving-cert\") pod \"648a2643-630c-4a9b-aac1-47351c37805b\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") "
Nov 03 18:39:26.744342 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744291    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-service-ca\") pod \"648a2643-630c-4a9b-aac1-47351c37805b\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") "
Nov 03 18:39:26.744342 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744320    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-serving-cert\") pod \"648a2643-630c-4a9b-aac1-47351c37805b\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") "
Nov 03 18:39:26.744486 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744351    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-oauth-config\") pod \"648a2643-630c-4a9b-aac1-47351c37805b\" (UID: \"648a2643-630c-4a9b-aac1-47351c37805b\") "
Nov 03 18:39:26.744539 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744521    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648a2643-630c-4a9b-aac1-47351c37805b-tmp" pod "648a2643-630c-4a9b-aac1-47351c37805b" (UID: "648a2643-630c-4a9b-aac1-47351c37805b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:26.744636 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744609    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-console-config" pod "648a2643-630c-4a9b-aac1-47351c37805b" (UID: "648a2643-630c-4a9b-aac1-47351c37805b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:26.744699 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744631    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-oauth-serving-cert" pod "648a2643-630c-4a9b-aac1-47351c37805b" (UID: "648a2643-630c-4a9b-aac1-47351c37805b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:26.744746 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.744687    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-service-ca" pod "648a2643-630c-4a9b-aac1-47351c37805b" (UID: "648a2643-630c-4a9b-aac1-47351c37805b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:39:26.746702 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.746675    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-oauth-config" pod "648a2643-630c-4a9b-aac1-47351c37805b" (UID: "648a2643-630c-4a9b-aac1-47351c37805b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:39:26.746802 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.746725    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-serving-cert" pod "648a2643-630c-4a9b-aac1-47351c37805b" (UID: "648a2643-630c-4a9b-aac1-47351c37805b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:39:26.746802 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.746736    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648a2643-630c-4a9b-aac1-47351c37805b-kube-api-access-9ms8f" pod "648a2643-630c-4a9b-aac1-47351c37805b" (UID: "648a2643-630c-4a9b-aac1-47351c37805b"). InnerVolumeSpecName "kube-api-access-9ms8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:26.845409 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.845378    4489 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-console-config\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:26.845409 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.845404    4489 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/648a2643-630c-4a9b-aac1-47351c37805b-tmp\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:26.845409 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.845414    4489 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-oauth-serving-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:26.845618 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.845422    4489 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/648a2643-630c-4a9b-aac1-47351c37805b-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:26.845618 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.845433    4489 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-serving-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:26.845618 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.845442    4489 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/648a2643-630c-4a9b-aac1-47351c37805b-console-oauth-config\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:26.845618 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:26.845452    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ms8f\" (UniqueName: \"kubernetes.io/projected/648a2643-630c-4a9b-aac1-47351c37805b-kube-api-access-9ms8f\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:39:27.356109 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.356034    4489 generic.go:356] "Generic (PLEG): container finished" podID="648a2643-630c-4a9b-aac1-47351c37805b" containerID="f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3" exitCode=0
Nov 03 18:39:27.356109 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.356068    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77898cc49b-qqkfs" event={"ID":"648a2643-630c-4a9b-aac1-47351c37805b","Type":"ContainerDied","Data":"f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3"}
Nov 03 18:39:27.356109 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.356078    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77898cc49b-qqkfs"
Nov 03 18:39:27.356109 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.356110    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77898cc49b-qqkfs" event={"ID":"648a2643-630c-4a9b-aac1-47351c37805b","Type":"ContainerDied","Data":"32ff578470c1cb0169a1c24857c1dc2b6f99c9df429a62c2ced41761ea37df84"}
Nov 03 18:39:27.356355 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.356130    4489 scope.go:122] "RemoveContainer" containerID="f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3"
Nov 03 18:39:27.368259 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.368237    4489 scope.go:122] "RemoveContainer" containerID="f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3"
Nov 03 18:39:27.368508 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:39:27.368487    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3\": container with ID starting with f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3 not found: ID does not exist" containerID="f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3"
Nov 03 18:39:27.368566 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.368520    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3"} err="failed to get container status \"f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3\": rpc error: code = NotFound desc = could not find container \"f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3\": container with ID starting with f69bd2af52dc1ae1fb50b21f89c0b4022b59c62f2e06523b232d75c05212bde3 not found: ID does not exist"
Nov 03 18:39:27.526312 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.526277    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77898cc49b-qqkfs"]
Nov 03 18:39:27.578296 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:27.578266    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77898cc49b-qqkfs"]
Nov 03 18:39:28.992473 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:28.992441    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648a2643-630c-4a9b-aac1-47351c37805b" path="/var/lib/kubelet/pods/648a2643-630c-4a9b-aac1-47351c37805b/volumes"
Nov 03 18:39:47.528673 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:39:47.528637    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:39:57.378348 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.378314    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7688575f-m685f"]
Nov 03 18:39:57.378843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.378539    4489 cpu_manager.go:414] "containerMap: removing container" podUID="648a2643-630c-4a9b-aac1-47351c37805b" containerName="console"
Nov 03 18:39:57.378843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.378549    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648a2643-630c-4a9b-aac1-47351c37805b" containerName="console"
Nov 03 18:39:57.378843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.378565    4489 cpu_manager.go:414] "containerMap: removing container" podUID="141aa4fe-227d-45f5-a995-1b20655868fd" containerName="insights-gathering"
Nov 03 18:39:57.378843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.378570    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141aa4fe-227d-45f5-a995-1b20655868fd" containerName="insights-gathering"
Nov 03 18:39:57.378843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.378639    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="648a2643-630c-4a9b-aac1-47351c37805b" containerName="console"
Nov 03 18:39:57.378843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.378648    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="141aa4fe-227d-45f5-a995-1b20655868fd" containerName="insights-gathering"
Nov 03 18:39:57.382717 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.382701    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.441737 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.441712    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-serving-cert\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.441882 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.441752    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-oauth-config\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.441882 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.441842    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-trusted-ca-bundle\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.441965 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.441904    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfg4k\" (UniqueName: \"kubernetes.io/projected/a804b15e-6eb7-4890-ab26-a84ccf41a706-kube-api-access-nfg4k\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.441965 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.441945    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-service-ca\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.441965 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.441962    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-config\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.442205 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.441994    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a804b15e-6eb7-4890-ab26-a84ccf41a706-tmp\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.442205 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.442085    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-oauth-serving-cert\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.479386 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.479361    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7688575f-m685f"]
Nov 03 18:39:57.543321 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543296    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-oauth-serving-cert\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543433 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543333    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-serving-cert\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543433 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543351    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-oauth-config\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543433 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543381    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-trusted-ca-bundle\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543433 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543395    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfg4k\" (UniqueName: \"kubernetes.io/projected/a804b15e-6eb7-4890-ab26-a84ccf41a706-kube-api-access-nfg4k\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543840 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543538    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-service-ca\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543840 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543578    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-config\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543840 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543609    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a804b15e-6eb7-4890-ab26-a84ccf41a706-tmp\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.543968 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.543925    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a804b15e-6eb7-4890-ab26-a84ccf41a706-tmp\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.544052 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.544036    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-oauth-serving-cert\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.544487 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.544461    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-service-ca\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.544645 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.544619    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-trusted-ca-bundle\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.544773 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.544755    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-config\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.546104 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.546081    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-serving-cert\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.546223 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.546081    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a804b15e-6eb7-4890-ab26-a84ccf41a706-console-oauth-config\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.609871 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.609851    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfg4k\" (UniqueName: \"kubernetes.io/projected/a804b15e-6eb7-4890-ab26-a84ccf41a706-kube-api-access-nfg4k\") pod \"console-7688575f-m685f\" (UID: \"a804b15e-6eb7-4890-ab26-a84ccf41a706\") " pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:57.694599 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:57.694577    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7688575f-m685f"
Nov 03 18:39:58.060080 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:39:58.060046    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda804b15e_6eb7_4890_ab26_a84ccf41a706.slice/crio-4e335e9ba4b09e161b7f5caab2f57484c57657df9e11f2cc595807f5f5334db1 WatchSource:0}: Error finding container 4e335e9ba4b09e161b7f5caab2f57484c57657df9e11f2cc595807f5f5334db1: Status 404 returned error can't find the container with id 4e335e9ba4b09e161b7f5caab2f57484c57657df9e11f2cc595807f5f5334db1
Nov 03 18:39:58.062972 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:58.062952    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7688575f-m685f"]
Nov 03 18:39:58.454854 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:58.454818    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7688575f-m685f" event={"ID":"a804b15e-6eb7-4890-ab26-a84ccf41a706","Type":"ContainerStarted","Data":"4b29dfba167e606a1b7f65fe39eacfeb90b74c95209a9adb070f438b4b1b5fbd"}
Nov 03 18:39:58.455270 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:58.454877    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7688575f-m685f" event={"ID":"a804b15e-6eb7-4890-ab26-a84ccf41a706","Type":"ContainerStarted","Data":"4e335e9ba4b09e161b7f5caab2f57484c57657df9e11f2cc595807f5f5334db1"}
Nov 03 18:39:58.577059 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:39:58.577004    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7688575f-m685f" podStartSLOduration=1.576991493 podStartE2EDuration="1.576991493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:39:58.576353445 +0000 UTC m=+127.725048015" watchObservedRunningTime="2026-11-03 18:39:58.576991493 +0000 UTC m=+127.725686133"
Nov 03 18:40:07.695329 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:07.695251    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7688575f-m685f"
Nov 03 18:40:07.695329 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:07.695307    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7688575f-m685f"
Nov 03 18:40:07.699919 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:07.699898    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7688575f-m685f"
Nov 03 18:40:08.496545 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:08.496511    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7688575f-m685f"
Nov 03 18:40:08.671788 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:08.671757    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6557849fc5-cpjm9"]
Nov 03 18:40:17.528605 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:40:17.528561    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:40:33.691294 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:33.691227    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6557849fc5-cpjm9" podUID="2884fc67-ce08-4844-bbfd-ed36f3a12a61" containerName="console" containerID="cri-o://f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5" gracePeriod=15
Nov 03 18:40:33.932434 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:33.932412    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:40:34.108988 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.108913    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-service-ca\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-service-ca\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.108988 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.108950    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-config\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-config\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.108988 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.108989    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2884fc67-ce08-4844-bbfd-ed36f3a12a61-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2884fc67-ce08-4844-bbfd-ed36f3a12a61-tmp\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.109307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109017    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-oauth-config\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.109307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109035    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-oauth-serving-cert\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.109307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109052    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-serving-cert\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.109307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109080    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2884fc67-ce08-4844-bbfd-ed36f3a12a61-kube-api-access-t5ggf\" (UniqueName: \"kubernetes.io/projected/2884fc67-ce08-4844-bbfd-ed36f3a12a61-kube-api-access-t5ggf\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.109307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109128    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-trusted-ca-bundle\") pod \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\" (UID: \"2884fc67-ce08-4844-bbfd-ed36f3a12a61\") "
Nov 03 18:40:34.109534 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109334    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2884fc67-ce08-4844-bbfd-ed36f3a12a61-tmp" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:40:34.109534 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109371    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-service-ca" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:34.109631 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109586    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-config" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:34.109759 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109660    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-trusted-ca-bundle" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:34.109989 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.109805    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-oauth-serving-cert" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:34.111413 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.111382    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-oauth-config" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:34.111541 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.111418    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-serving-cert" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:34.111541 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.111470    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2884fc67-ce08-4844-bbfd-ed36f3a12a61-kube-api-access-t5ggf" pod "2884fc67-ce08-4844-bbfd-ed36f3a12a61" (UID: "2884fc67-ce08-4844-bbfd-ed36f3a12a61"). InnerVolumeSpecName "kube-api-access-t5ggf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:40:34.210464 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210431    4489 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.210464 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210460    4489 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-config\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.210638 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210472    4489 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2884fc67-ce08-4844-bbfd-ed36f3a12a61-tmp\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.210638 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210486    4489 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-oauth-config\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.210638 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210498    4489 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-oauth-serving-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.210638 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210509    4489 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2884fc67-ce08-4844-bbfd-ed36f3a12a61-console-serving-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.210638 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210521    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5ggf\" (UniqueName: \"kubernetes.io/projected/2884fc67-ce08-4844-bbfd-ed36f3a12a61-kube-api-access-t5ggf\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.210638 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.210535    4489 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2884fc67-ce08-4844-bbfd-ed36f3a12a61-trusted-ca-bundle\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:40:34.577385 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.577352    4489 generic.go:356] "Generic (PLEG): container finished" podID="2884fc67-ce08-4844-bbfd-ed36f3a12a61" containerID="f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5" exitCode=0
Nov 03 18:40:34.577533 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.577408    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6557849fc5-cpjm9"
Nov 03 18:40:34.577533 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.577429    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6557849fc5-cpjm9" event={"ID":"2884fc67-ce08-4844-bbfd-ed36f3a12a61","Type":"ContainerDied","Data":"f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5"}
Nov 03 18:40:34.577533 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.577488    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6557849fc5-cpjm9" event={"ID":"2884fc67-ce08-4844-bbfd-ed36f3a12a61","Type":"ContainerDied","Data":"1e849e3a6f5d90319c614c4d5f78c6d2b1e14c331164150597e392b00b309d81"}
Nov 03 18:40:34.577533 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.577507    4489 scope.go:122] "RemoveContainer" containerID="f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5"
Nov 03 18:40:34.585756 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.585740    4489 scope.go:122] "RemoveContainer" containerID="f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5"
Nov 03 18:40:34.586023 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:40:34.586005    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5\": container with ID starting with f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5 not found: ID does not exist" containerID="f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5"
Nov 03 18:40:34.586089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.586034    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5"} err="failed to get container status \"f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5\": rpc error: code = NotFound desc = could not find container \"f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5\": container with ID starting with f5b5c9484b88a2d65a878b0c2c4234c3143bbc0e9fc7d526f4efbb9927b6f9f5 not found: ID does not exist"
Nov 03 18:40:34.747156 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.747118    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6557849fc5-cpjm9"]
Nov 03 18:40:34.799037 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.799010    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6557849fc5-cpjm9"]
Nov 03 18:40:34.991860 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:40:34.991831    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2884fc67-ce08-4844-bbfd-ed36f3a12a61" path="/var/lib/kubelet/pods/2884fc67-ce08-4844-bbfd-ed36f3a12a61/volumes"
Nov 03 18:40:47.527680 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:40:47.527644    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:41:17.528733 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:41:17.528695    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:41:47.528752 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:41:47.528719    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:42:17.528969 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:42:17.528935    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:42:47.527819 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:42:47.527781    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:43:17.528605 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:43:17.528527    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:43:30.693868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.693830    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"]
Nov 03 18:43:30.694248 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.694080    4489 cpu_manager.go:414] "containerMap: removing container" podUID="2884fc67-ce08-4844-bbfd-ed36f3a12a61" containerName="console"
Nov 03 18:43:30.694248 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.694092    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2884fc67-ce08-4844-bbfd-ed36f3a12a61" containerName="console"
Nov 03 18:43:30.694248 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.694136    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="2884fc67-ce08-4844-bbfd-ed36f3a12a61" containerName="console"
Nov 03 18:43:30.696832 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.696815    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.754308 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.754283    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 18:43:30.754479 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.754294    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 18:43:30.754479 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.754294    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 18:43:30.754612 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.754541    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 18:43:30.795446 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.795414    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.795446 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.795445    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.795609 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.795466    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.795609 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.795562    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.795609 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.795596    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ccac84-0f9c-4ac3-888f-5948c016d079-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.795715 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.795659    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.795715 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.795690    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdshg\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-kube-api-access-kdshg\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.806109 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.806088    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4qf6p\""
Nov 03 18:43:30.806426 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.806413    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 18:43:30.806426 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.806415    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 18:43:30.807255 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.807240    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"]
Nov 03 18:43:30.896830 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.896803    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdshg\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-kube-api-access-kdshg\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.896966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.896836    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.896966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.896853    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.896966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.896870    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.896966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.896897    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.896966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.896941    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ccac84-0f9c-4ac3-888f-5948c016d079-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.897265 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.896973    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.897725 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.897607    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.899374 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.899353    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.899473 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.899417    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.899659 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.899641    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.899659 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.899646    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ccac84-0f9c-4ac3-888f-5948c016d079-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.960020 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.959969    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:30.960175 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:30.960133    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdshg\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-kube-api-access-kdshg\") pod \"istiod-openshift-gateway-7876bfcb74-p2sxw\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:31.008129 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:31.008107    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:31.423175 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:43:31.423128    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ccac84_0f9c_4ac3_888f_5948c016d079.slice/crio-91e608693852b2f4572303e5c936ec343f336ea10d482ec0ca32673f7df12e75 WatchSource:0}: Error finding container 91e608693852b2f4572303e5c936ec343f336ea10d482ec0ca32673f7df12e75: Status 404 returned error can't find the container with id 91e608693852b2f4572303e5c936ec343f336ea10d482ec0ca32673f7df12e75
Nov 03 18:43:31.425156 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:31.425123    4489 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:43:31.425686 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:31.425666    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"]
Nov 03 18:43:32.116371 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:32.116334    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" event={"ID":"90ccac84-0f9c-4ac3-888f-5948c016d079","Type":"ContainerStarted","Data":"91e608693852b2f4572303e5c936ec343f336ea10d482ec0ca32673f7df12e75"}
Nov 03 18:43:35.323782 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:35.323752    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:43:35.324051 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:35.323814    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:43:36.131249 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:36.131218    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" event={"ID":"90ccac84-0f9c-4ac3-888f-5948c016d079","Type":"ContainerStarted","Data":"d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398"}
Nov 03 18:43:36.131425 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:36.131374    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:36.132863 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:36.132837    4489 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-p2sxw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:43:36.132974 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:36.132882    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" podUID="90ccac84-0f9c-4ac3-888f-5948c016d079" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:43:36.348800 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:36.348750    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" podStartSLOduration=2.451011116 podStartE2EDuration="6.34873711s" totalImagesPullingTime="3.897725994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:36.308665818 +0000 UTC m=+345.457360390" watchObservedRunningTime="2026-11-03 18:43:36.34873711 +0000 UTC m=+345.497431681"
Nov 03 18:43:37.135637 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:37.135604    4489 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-p2sxw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:43:37.135778 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:37.135652    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" podUID="90ccac84-0f9c-4ac3-888f-5948c016d079" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:43:40.136006 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:43:40.135975    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:43:47.528802 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:43:47.528762    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:44:17.528183 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:44:17.528126    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:44:30.518447 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.518412    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"]
Nov 03 18:44:30.523800 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.523779    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.624596 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.624564    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ce114c-7ba4-48e1-b680-d81ac2448390-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.624596 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.624601    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrmd\" (UniqueName: \"kubernetes.io/projected/c0ce114c-7ba4-48e1-b680-d81ac2448390-kube-api-access-4lrmd\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.624797 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.624631    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.624797 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.624744    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ce114c-7ba4-48e1-b680-d81ac2448390-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.624797 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.624764    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.624797 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.624779    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.641185 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.641159    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"]
Nov 03 18:44:30.656156 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.656117    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 18:44:30.656291 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.656176    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 18:44:30.656291 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.656198    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jdkcr\""
Nov 03 18:44:30.656291 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.656224    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 18:44:30.656291 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.656226    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 18:44:30.656291 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.656248    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 18:44:30.725731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.725698    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ce114c-7ba4-48e1-b680-d81ac2448390-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.725912 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.725746    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmd\" (UniqueName: \"kubernetes.io/projected/c0ce114c-7ba4-48e1-b680-d81ac2448390-kube-api-access-4lrmd\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.725912 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.725870    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.725991 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.725958    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ce114c-7ba4-48e1-b680-d81ac2448390-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.726024 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.725995    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.726064 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.726021    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.726238 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.726219    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ce114c-7ba4-48e1-b680-d81ac2448390-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.728469 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.728444    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ce114c-7ba4-48e1-b680-d81ac2448390-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.728720 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.728700    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.728720 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.728711    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.728833 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.728775    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0ce114c-7ba4-48e1-b680-d81ac2448390-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.789368 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.789301    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrmd\" (UniqueName: \"kubernetes.io/projected/c0ce114c-7ba4-48e1-b680-d81ac2448390-kube-api-access-4lrmd\") pod \"opendatahub-operator-controller-manager-7b94cbc577-jjmrx\" (UID: \"c0ce114c-7ba4-48e1-b680-d81ac2448390\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:30.836087 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:30.836057    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:31.360322 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:44:31.360296    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ce114c_7ba4_48e1_b680_d81ac2448390.slice/crio-c4a5afdf6052e87e83fcc68960871f19155fd5cdc3f4cc87b3f3e5f181b52cdf WatchSource:0}: Error finding container c4a5afdf6052e87e83fcc68960871f19155fd5cdc3f4cc87b3f3e5f181b52cdf: Status 404 returned error can't find the container with id c4a5afdf6052e87e83fcc68960871f19155fd5cdc3f4cc87b3f3e5f181b52cdf
Nov 03 18:44:31.362778 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:31.362751    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"]
Nov 03 18:44:32.305678 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:32.305639    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx" event={"ID":"c0ce114c-7ba4-48e1-b680-d81ac2448390","Type":"ContainerStarted","Data":"c4a5afdf6052e87e83fcc68960871f19155fd5cdc3f4cc87b3f3e5f181b52cdf"}
Nov 03 18:44:36.321636 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:36.321600    4489 generic.go:356] "Generic (PLEG): container finished" podID="c0ce114c-7ba4-48e1-b680-d81ac2448390" containerID="a532ff52744cd09eb7185239dcc67e0563a887b3124e63325072b2b5170703da" exitCode=0
Nov 03 18:44:36.322101 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:36.321676    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx" event={"ID":"c0ce114c-7ba4-48e1-b680-d81ac2448390","Type":"ContainerDied","Data":"a532ff52744cd09eb7185239dcc67e0563a887b3124e63325072b2b5170703da"}
Nov 03 18:44:37.327089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:37.327054    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx" event={"ID":"c0ce114c-7ba4-48e1-b680-d81ac2448390","Type":"ContainerStarted","Data":"627155e1bc08ff94682a6e3e987990a6dd1b41d54e59758f3ce8a67cc8a1794b"}
Nov 03 18:44:37.327535 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:37.327245    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:44:37.575727 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:37.575672    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx" podStartSLOduration=2.325548758 podStartE2EDuration="7.575656781s" totalImagesPullingTime="4.780492246s" totalInitContainerRuntime="469.615777ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:37.523935486 +0000 UTC m=+406.672630058" watchObservedRunningTime="2026-11-03 18:44:37.575656781 +0000 UTC m=+406.724351351"
Nov 03 18:44:47.527884 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:44:47.527849    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:44:48.333655 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:44:48.333627    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-jjmrx"
Nov 03 18:45:04.433299 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.433266    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"]
Nov 03 18:45:04.435496 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.435477    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.494713 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.494684    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk7bl\" (UniqueName: \"kubernetes.io/projected/5d350369-34fb-41cd-b1dd-11dfcaabef4f-kube-api-access-nk7bl\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.494873 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.494757    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d350369-34fb-41cd-b1dd-11dfcaabef4f-tls-certs\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.494873 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.494818    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d350369-34fb-41cd-b1dd-11dfcaabef4f-tmp\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.543505 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.543471    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"]
Nov 03 18:45:04.594553 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.594524    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 18:45:04.595807 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.595786    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk7bl\" (UniqueName: \"kubernetes.io/projected/5d350369-34fb-41cd-b1dd-11dfcaabef4f-kube-api-access-nk7bl\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.595864 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.595818    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d350369-34fb-41cd-b1dd-11dfcaabef4f-tls-certs\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.595960 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.595946    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d350369-34fb-41cd-b1dd-11dfcaabef4f-tmp\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.598299 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.598271    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d350369-34fb-41cd-b1dd-11dfcaabef4f-tmp\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.606634 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.606608    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kgrlf\""
Nov 03 18:45:04.606634 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.606608    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 18:45:04.619137 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.619115    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d350369-34fb-41cd-b1dd-11dfcaabef4f-tls-certs\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.658867 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.658836    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk7bl\" (UniqueName: \"kubernetes.io/projected/5d350369-34fb-41cd-b1dd-11dfcaabef4f-kube-api-access-nk7bl\") pod \"kube-auth-proxy-b8f876b79-4bl78\" (UID: \"5d350369-34fb-41cd-b1dd-11dfcaabef4f\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:04.746360 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:04.746264    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"
Nov 03 18:45:05.118456 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:05.118408    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8f876b79-4bl78"]
Nov 03 18:45:05.426776 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:05.426744    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78" event={"ID":"5d350369-34fb-41cd-b1dd-11dfcaabef4f","Type":"ContainerStarted","Data":"450ab854685c7e778181a12e97befefd0d204ee7d8f5d0f4aa6fbcec93e0df3c"}
Nov 03 18:45:08.440481 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:08.440442    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78" event={"ID":"5d350369-34fb-41cd-b1dd-11dfcaabef4f","Type":"ContainerStarted","Data":"60d557b5a85887ba24122173b3965f51d2681bdd5e431cb88adeffaba43369b1"}
Nov 03 18:45:08.569603 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:08.569552    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b8f876b79-4bl78" podStartSLOduration=1.728299245 podStartE2EDuration="4.569538805s" totalImagesPullingTime="2.84123956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:08.569078957 +0000 UTC m=+437.717773546" watchObservedRunningTime="2026-11-03 18:45:08.569538805 +0000 UTC m=+437.718233376"
Nov 03 18:45:16.087802 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.087760    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"]
Nov 03 18:45:16.090808 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.090785    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.145199 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.145165    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"]
Nov 03 18:45:16.191686 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.191651    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1deac7e7-8a39-45db-80fd-7ef84c90b418-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.191848 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.191694    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1deac7e7-8a39-45db-80fd-7ef84c90b418-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.191848 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.191713    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w7q\" (UniqueName: \"kubernetes.io/projected/1deac7e7-8a39-45db-80fd-7ef84c90b418-kube-api-access-89w7q\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.196126 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.196098    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 18:45:16.197641 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.197624    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4z4fj\""
Nov 03 18:45:16.197752 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.197736    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 18:45:16.197800 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.197738    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 18:45:16.197800 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.197769    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 18:45:16.293209 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.293176    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1deac7e7-8a39-45db-80fd-7ef84c90b418-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.293209 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.293222    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1deac7e7-8a39-45db-80fd-7ef84c90b418-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.293464 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.293242    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w7q\" (UniqueName: \"kubernetes.io/projected/1deac7e7-8a39-45db-80fd-7ef84c90b418-kube-api-access-89w7q\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.293817 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.293789    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1deac7e7-8a39-45db-80fd-7ef84c90b418-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.295788 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.295764    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1deac7e7-8a39-45db-80fd-7ef84c90b418-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.355944 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.355870    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w7q\" (UniqueName: \"kubernetes.io/projected/1deac7e7-8a39-45db-80fd-7ef84c90b418-kube-api-access-89w7q\") pod \"kuadrant-console-plugin-66c4fb575d-tjbps\" (UID: \"1deac7e7-8a39-45db-80fd-7ef84c90b418\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.410783 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.410748    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"
Nov 03 18:45:16.779717 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:45:16.779679    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1deac7e7_8a39_45db_80fd_7ef84c90b418.slice/crio-d07f79d0ec2076e0c2ca791b3bebeeed87805c40cf8e8a597a55afb1a7eeab20 WatchSource:0}: Error finding container d07f79d0ec2076e0c2ca791b3bebeeed87805c40cf8e8a597a55afb1a7eeab20: Status 404 returned error can't find the container with id d07f79d0ec2076e0c2ca791b3bebeeed87805c40cf8e8a597a55afb1a7eeab20
Nov 03 18:45:16.781969 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:16.781946    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps"]
Nov 03 18:45:17.472162 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:17.472117    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps" event={"ID":"1deac7e7-8a39-45db-80fd-7ef84c90b418","Type":"ContainerStarted","Data":"d07f79d0ec2076e0c2ca791b3bebeeed87805c40cf8e8a597a55afb1a7eeab20"}
Nov 03 18:45:17.530418 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:45:17.530383    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:45:24.501290 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:24.501251    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps" event={"ID":"1deac7e7-8a39-45db-80fd-7ef84c90b418","Type":"ContainerStarted","Data":"b67836d9162373024a33262d8af8148f1a7d59a76c4528ee74a7a4498f056a83"}
Nov 03 18:45:24.624179 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:45:24.624069    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-tjbps" podStartSLOduration=1.068538329 podStartE2EDuration="8.624054278s" totalImagesPullingTime="7.555515949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:24.622896955 +0000 UTC m=+453.771591536" watchObservedRunningTime="2026-11-03 18:45:24.624054278 +0000 UTC m=+453.772748843"
Nov 03 18:45:47.529403 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:45:47.529363    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:46:04.793703 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.793623    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"]
Nov 03 18:46:04.796898 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.796881    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.852914 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.852884    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"]
Nov 03 18:46:04.868368 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.868338    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.868527 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.868372    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvfd\" (UniqueName: \"kubernetes.io/projected/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-kube-api-access-gxvfd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.868527 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.868454    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.868527 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.868514    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.903771 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.903737    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 18:46:04.903771 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.903745    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 18:46:04.904017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.903775    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-rh2ts\""
Nov 03 18:46:04.904017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.903796    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 18:46:04.904017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.903811    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 18:46:04.904017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.903897    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 18:46:04.969070 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.969029    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.969274 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.969086    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvfd\" (UniqueName: \"kubernetes.io/projected/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-kube-api-access-gxvfd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.969274 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.969136    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.969274 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.969203    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:04.969445 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:04.969311    4489 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 18:46:04.969522 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:04.969503    4489 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 18:46:04.969608 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:04.969590    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-proxy-source podName:2c4f42a8-138c-48c0-8e51-75f7d4a1e302 nodeName:}" failed. No retries permitted until 2026-11-03 18:46:05.469571225 +0000 UTC m=+494.618265773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-q5hdm" (UID: "2c4f42a8-138c-48c0-8e51-75f7d4a1e302") : configmap "usage-tenancy-proxy-source" not found
Nov 03 18:46:04.969681 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:04.969618    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-tls-cert podName:2c4f42a8-138c-48c0-8e51-75f7d4a1e302 nodeName:}" failed. No retries permitted until 2026-11-03 18:46:05.469609514 +0000 UTC m=+494.618304068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-q5hdm" (UID: "2c4f42a8-138c-48c0-8e51-75f7d4a1e302") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 18:46:04.969843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:04.969810    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:05.034943 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:05.034912    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvfd\" (UniqueName: \"kubernetes.io/projected/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-kube-api-access-gxvfd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:05.471914 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:05.471876    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:05.471914 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:05.471919    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:05.472753 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:05.472687    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:05.474689 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:05.474660    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c4f42a8-138c-48c0-8e51-75f7d4a1e302-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5hdm\" (UID: \"2c4f42a8-138c-48c0-8e51-75f7d4a1e302\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:05.709656 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:05.709617    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:06.078641 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:46:06.078608    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4f42a8_138c_48c0_8e51_75f7d4a1e302.slice/crio-ef2eca6a1e3f648019e25461da8cffa639e3a154f376070bcd88e626ad93e98c WatchSource:0}: Error finding container ef2eca6a1e3f648019e25461da8cffa639e3a154f376070bcd88e626ad93e98c: Status 404 returned error can't find the container with id ef2eca6a1e3f648019e25461da8cffa639e3a154f376070bcd88e626ad93e98c
Nov 03 18:46:06.081200 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:06.081180    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"]
Nov 03 18:46:06.641223 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:06.641180    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm" event={"ID":"2c4f42a8-138c-48c0-8e51-75f7d4a1e302","Type":"ContainerStarted","Data":"ef2eca6a1e3f648019e25461da8cffa639e3a154f376070bcd88e626ad93e98c"}
Nov 03 18:46:17.531787 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:17.531730    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:46:21.702934 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:21.702884    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm" event={"ID":"2c4f42a8-138c-48c0-8e51-75f7d4a1e302","Type":"ContainerStarted","Data":"c732d9f72b84c702b15f3a49aaee789adb5d0f84781b3b30cb988a4a1d889649"}
Nov 03 18:46:21.703410 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:21.703030    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:21.842626 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:21.842571    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm" podStartSLOduration=2.533280446 podStartE2EDuration="17.842553999s" totalImagesPullingTime="15.309273553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:21.841727931 +0000 UTC m=+510.990422502" watchObservedRunningTime="2026-11-03 18:46:21.842553999 +0000 UTC m=+510.991248573"
Nov 03 18:46:30.898781 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:30.898736    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"]
Nov 03 18:46:30.902039 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:30.902014    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:30.964337 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:30.964308    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"]
Nov 03 18:46:30.989481 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:30.989443    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f906b51-e890-4255-907a-796b8930ad5d-cert\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:30.989642 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:30.989501    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f906b51-e890-4255-907a-796b8930ad5d-metrics-certs\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:30.989642 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:30.989606    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f906b51-e890-4255-907a-796b8930ad5d-manager-config\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:30.989731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:30.989647    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79j9\" (UniqueName: \"kubernetes.io/projected/1f906b51-e890-4255-907a-796b8930ad5d-kube-api-access-c79j9\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.015281 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.015250    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 18:46:31.015452 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.015290    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:46:31.015452 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.015316    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 18:46:31.015574 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.015333    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-bbwk5\""
Nov 03 18:46:31.015574 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.015337    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 18:46:31.015574 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.015381    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 18:46:31.091048 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.091002    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f906b51-e890-4255-907a-796b8930ad5d-cert\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.091248 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.091083    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f906b51-e890-4255-907a-796b8930ad5d-metrics-certs\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.091248 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.091167    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f906b51-e890-4255-907a-796b8930ad5d-manager-config\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.091248 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.091211    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79j9\" (UniqueName: \"kubernetes.io/projected/1f906b51-e890-4255-907a-796b8930ad5d-kube-api-access-c79j9\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.092136 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.092107    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f906b51-e890-4255-907a-796b8930ad5d-manager-config\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.093863 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.093835    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f906b51-e890-4255-907a-796b8930ad5d-metrics-certs\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.093954 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.093880    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f906b51-e890-4255-907a-796b8930ad5d-cert\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.160924 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.160894    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79j9\" (UniqueName: \"kubernetes.io/projected/1f906b51-e890-4255-907a-796b8930ad5d-kube-api-access-c79j9\") pod \"jobset-controller-manager-74f448fddc-8h92r\" (UID: \"1f906b51-e890-4255-907a-796b8930ad5d\") " pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.213831 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.213791    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:31.597077 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:46:31.597040    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f906b51_e890_4255_907a_796b8930ad5d.slice/crio-a8bcf000f5b67629ce8bf62962017740f9305c53d26a2a1ae5f7508562034e6a WatchSource:0}: Error finding container a8bcf000f5b67629ce8bf62962017740f9305c53d26a2a1ae5f7508562034e6a: Status 404 returned error can't find the container with id a8bcf000f5b67629ce8bf62962017740f9305c53d26a2a1ae5f7508562034e6a
Nov 03 18:46:31.598517 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.598454    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"]
Nov 03 18:46:31.742182 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:31.742133    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r" event={"ID":"1f906b51-e890-4255-907a-796b8930ad5d","Type":"ContainerStarted","Data":"a8bcf000f5b67629ce8bf62962017740f9305c53d26a2a1ae5f7508562034e6a"}
Nov 03 18:46:32.712916 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:32.712879    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5hdm"
Nov 03 18:46:36.764325 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:36.764234    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r" event={"ID":"1f906b51-e890-4255-907a-796b8930ad5d","Type":"ContainerStarted","Data":"7dc431c5f410028843537afad6c496d08301a32efd5b8e65484c1b55001e8bf8"}
Nov 03 18:46:36.764841 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:36.764374    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:36.884668 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:36.884615    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r" podStartSLOduration=2.019347007 podStartE2EDuration="6.88460186s" totalImagesPullingTime="4.865254853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:36.884274975 +0000 UTC m=+526.032969547" watchObservedRunningTime="2026-11-03 18:46:36.88460186 +0000 UTC m=+526.033296428"
Nov 03 18:46:41.986478 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:41.986440    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"]
Nov 03 18:46:41.986923 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:41.986633    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" podUID="90ccac84-0f9c-4ac3-888f-5948c016d079" containerName="discovery" containerID="cri-o://d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398" gracePeriod=30
Nov 03 18:46:42.233406 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.233379    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:46:42.384469 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.384367    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-cacerts\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-cacerts\") pod \"90ccac84-0f9c-4ac3-888f-5948c016d079\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") "
Nov 03 18:46:42.384469 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.384421    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-kubeconfig\") pod \"90ccac84-0f9c-4ac3-888f-5948c016d079\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") "
Nov 03 18:46:42.384469 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.384452    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-dns-cert\") pod \"90ccac84-0f9c-4ac3-888f-5948c016d079\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") "
Nov 03 18:46:42.384767 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.384745    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-ca-configmap\") pod \"90ccac84-0f9c-4ac3-888f-5948c016d079\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") "
Nov 03 18:46:42.384906 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.384880    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-token\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-token\") pod \"90ccac84-0f9c-4ac3-888f-5948c016d079\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") "
Nov 03 18:46:42.385033 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.384949    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-kube-api-access-kdshg\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-kube-api-access-kdshg\") pod \"90ccac84-0f9c-4ac3-888f-5948c016d079\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") "
Nov 03 18:46:42.386637 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.386604    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-ca-configmap" pod "90ccac84-0f9c-4ac3-888f-5948c016d079" (UID: "90ccac84-0f9c-4ac3-888f-5948c016d079"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:46:42.387681 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.387651    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ccac84-0f9c-4ac3-888f-5948c016d079-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ccac84-0f9c-4ac3-888f-5948c016d079-local-certs\") pod \"90ccac84-0f9c-4ac3-888f-5948c016d079\" (UID: \"90ccac84-0f9c-4ac3-888f-5948c016d079\") "
Nov 03 18:46:42.387966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.387944    4489 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-ca-configmap\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:42.388110 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.387966    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-kubeconfig" pod "90ccac84-0f9c-4ac3-888f-5948c016d079" (UID: "90ccac84-0f9c-4ac3-888f-5948c016d079"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:46:42.388110 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.387964    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-cacerts" pod "90ccac84-0f9c-4ac3-888f-5948c016d079" (UID: "90ccac84-0f9c-4ac3-888f-5948c016d079"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:46:42.388356 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.388014    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-dns-cert" pod "90ccac84-0f9c-4ac3-888f-5948c016d079" (UID: "90ccac84-0f9c-4ac3-888f-5948c016d079"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:46:42.388879 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.388852    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-token" pod "90ccac84-0f9c-4ac3-888f-5948c016d079" (UID: "90ccac84-0f9c-4ac3-888f-5948c016d079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:46:42.389297 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.389276    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-kube-api-access-kdshg" pod "90ccac84-0f9c-4ac3-888f-5948c016d079" (UID: "90ccac84-0f9c-4ac3-888f-5948c016d079"). InnerVolumeSpecName "kube-api-access-kdshg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:46:42.389800 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.389784    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ccac84-0f9c-4ac3-888f-5948c016d079-local-certs" pod "90ccac84-0f9c-4ac3-888f-5948c016d079" (UID: "90ccac84-0f9c-4ac3-888f-5948c016d079"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:46:42.489441 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.489396    4489 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-cacerts\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:42.489441 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.489434    4489 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-kubeconfig\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:42.489441 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.489447    4489 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-csr-dns-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:42.489696 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.489459    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:42.489696 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.489472    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdshg\" (UniqueName: \"kubernetes.io/projected/90ccac84-0f9c-4ac3-888f-5948c016d079-kube-api-access-kdshg\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:42.489696 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.489487    4489 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ccac84-0f9c-4ac3-888f-5948c016d079-local-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:42.786457 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.786419    4489 generic.go:356] "Generic (PLEG): container finished" podID="90ccac84-0f9c-4ac3-888f-5948c016d079" containerID="d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398" exitCode=0
Nov 03 18:46:42.786646 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.786465    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" event={"ID":"90ccac84-0f9c-4ac3-888f-5948c016d079","Type":"ContainerDied","Data":"d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398"}
Nov 03 18:46:42.786646 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.786473    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"
Nov 03 18:46:42.786646 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.786551    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw" event={"ID":"90ccac84-0f9c-4ac3-888f-5948c016d079","Type":"ContainerDied","Data":"91e608693852b2f4572303e5c936ec343f336ea10d482ec0ca32673f7df12e75"}
Nov 03 18:46:42.786646 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.786580    4489 scope.go:122] "RemoveContainer" containerID="d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398"
Nov 03 18:46:42.795525 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.795505    4489 scope.go:122] "RemoveContainer" containerID="d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398"
Nov 03 18:46:42.795845 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:42.795826    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398\": container with ID starting with d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398 not found: ID does not exist" containerID="d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398"
Nov 03 18:46:42.795900 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.795853    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398"} err="failed to get container status \"d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398\": rpc error: code = NotFound desc = could not find container \"d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398\": container with ID starting with d62387539ddae7a31ec8dfae29a8d0f1c2aa4e96537c218521646bea4a02b398 not found: ID does not exist"
Nov 03 18:46:42.960233 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.960199    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"]
Nov 03 18:46:42.969801 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.969777    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p2sxw"]
Nov 03 18:46:42.993390 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:42.993364    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ccac84-0f9c-4ac3-888f-5948c016d079" path="/var/lib/kubelet/pods/90ccac84-0f9c-4ac3-888f-5948c016d079/volumes"
Nov 03 18:46:47.529482 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:47.529445    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:46:47.773715 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:47.773687    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-74f448fddc-8h92r"
Nov 03 18:46:49.220754 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.220724    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bfk8g"]
Nov 03 18:46:49.221136 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.220978    4489 cpu_manager.go:414] "containerMap: removing container" podUID="90ccac84-0f9c-4ac3-888f-5948c016d079" containerName="discovery"
Nov 03 18:46:49.221136 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.220988    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ccac84-0f9c-4ac3-888f-5948c016d079" containerName="discovery"
Nov 03 18:46:49.221136 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.221048    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ccac84-0f9c-4ac3-888f-5948c016d079" containerName="discovery"
Nov 03 18:46:49.479853 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.479779    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bfk8g"]
Nov 03 18:46:49.479853 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.479811    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"]
Nov 03 18:46:49.480034 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.479902    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bfk8g"
Nov 03 18:46:49.495250 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.495228    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"]
Nov 03 18:46:49.495351 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.495328    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.534675 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.534647    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2q9wr\""
Nov 03 18:46:49.586619 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.586575    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:46:49.586811 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.586788    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:46:49.587878 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.587855    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bfk8g"]
Nov 03 18:46:49.588072 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:49.588051    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-658dw], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-bfk8g" podUID="936e953f-9d94-4cf0-938d-0f3692825d79"
Nov 03 18:46:49.610528 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.610494    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68c48687b6-cszvm"]
Nov 03 18:46:49.628290 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.628265    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c48687b6-cszvm"
Nov 03 18:46:49.638788 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.638771    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 18:46:49.638947 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.638934    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 18:46:49.648267 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.648250    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\""
Nov 03 18:46:49.648494 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.648477    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-maas-api-tls\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.648532 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.648517    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-metrics-certs\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.648564 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.648538    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrjf\" (UniqueName: \"kubernetes.io/projected/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-kube-api-access-sdrjf\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.648627 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.648610    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-openshift-service-ca\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.648702 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.648688    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-658dw\" (UniqueName: \"kubernetes.io/projected/936e953f-9d94-4cf0-938d-0f3692825d79-kube-api-access-658dw\") pod \"authorino-6f6944bcdd-bfk8g\" (UID: \"936e953f-9d94-4cf0-938d-0f3692825d79\") " pod="kuadrant-system/authorino-6f6944bcdd-bfk8g"
Nov 03 18:46:49.666693 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.666670    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68c48687b6-cszvm"]
Nov 03 18:46:49.750110 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.750021    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-658dw\" (UniqueName: \"kubernetes.io/projected/936e953f-9d94-4cf0-938d-0f3692825d79-kube-api-access-658dw\") pod \"authorino-6f6944bcdd-bfk8g\" (UID: \"936e953f-9d94-4cf0-938d-0f3692825d79\") " pod="kuadrant-system/authorino-6f6944bcdd-bfk8g"
Nov 03 18:46:49.750110 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.750074    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-maas-api-tls\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.750360 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.750118    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-metrics-certs\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.750360 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.750163    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrjf\" (UniqueName: \"kubernetes.io/projected/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-kube-api-access-sdrjf\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.750360 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.750190    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2ld\" (UniqueName: \"kubernetes.io/projected/5dae7546-f63e-49b3-9881-ee51a985942e-kube-api-access-gg2ld\") pod \"authorino-68c48687b6-cszvm\" (UID: \"5dae7546-f63e-49b3-9881-ee51a985942e\") " pod="kuadrant-system/authorino-68c48687b6-cszvm"
Nov 03 18:46:49.750360 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.750218    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-openshift-service-ca\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.750914 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.750891    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-openshift-service-ca\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.752678 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.752658    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-maas-api-tls\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.752949 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.752927    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-metrics-certs\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.812613 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.812575    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bfk8g"
Nov 03 18:46:49.813629 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.813598    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-658dw\" (UniqueName: \"kubernetes.io/projected/936e953f-9d94-4cf0-938d-0f3692825d79-kube-api-access-658dw\") pod \"authorino-6f6944bcdd-bfk8g\" (UID: \"936e953f-9d94-4cf0-938d-0f3692825d79\") " pod="kuadrant-system/authorino-6f6944bcdd-bfk8g"
Nov 03 18:46:49.813629 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.813616    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrjf\" (UniqueName: \"kubernetes.io/projected/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-kube-api-access-sdrjf\") pod \"maas-api-54b47b974d-bj2h4\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") " pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.831361 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.831333    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:49.850945 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.850914    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2ld\" (UniqueName: \"kubernetes.io/projected/5dae7546-f63e-49b3-9881-ee51a985942e-kube-api-access-gg2ld\") pod \"authorino-68c48687b6-cszvm\" (UID: \"5dae7546-f63e-49b3-9881-ee51a985942e\") " pod="kuadrant-system/authorino-68c48687b6-cszvm"
Nov 03 18:46:49.858825 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.858807    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bfk8g"
Nov 03 18:46:49.914052 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.914020    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2ld\" (UniqueName: \"kubernetes.io/projected/5dae7546-f63e-49b3-9881-ee51a985942e-kube-api-access-gg2ld\") pod \"authorino-68c48687b6-cszvm\" (UID: \"5dae7546-f63e-49b3-9881-ee51a985942e\") " pod="kuadrant-system/authorino-68c48687b6-cszvm"
Nov 03 18:46:49.950581 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.950546    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c48687b6-cszvm"
Nov 03 18:46:49.951558 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.951523    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e953f-9d94-4cf0-938d-0f3692825d79-kube-api-access-658dw\" (UniqueName: \"kubernetes.io/projected/936e953f-9d94-4cf0-938d-0f3692825d79-kube-api-access-658dw\") pod \"936e953f-9d94-4cf0-938d-0f3692825d79\" (UID: \"936e953f-9d94-4cf0-938d-0f3692825d79\") "
Nov 03 18:46:49.953788 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:49.953760    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e953f-9d94-4cf0-938d-0f3692825d79-kube-api-access-658dw" pod "936e953f-9d94-4cf0-938d-0f3692825d79" (UID: "936e953f-9d94-4cf0-938d-0f3692825d79"). InnerVolumeSpecName "kube-api-access-658dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:46:50.046480 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.046448    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68c48687b6-cszvm"]
Nov 03 18:46:50.053073 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.053045    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-658dw\" (UniqueName: \"kubernetes.io/projected/936e953f-9d94-4cf0-938d-0f3692825d79-kube-api-access-658dw\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:50.078687 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.078659    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6576df7764-h75vj"]
Nov 03 18:46:50.105454 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.105429    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.136326 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.136288    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6576df7764-h75vj"]
Nov 03 18:46:50.159961 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.159936    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 18:46:50.240317 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:46:50.240291    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f9a639_2fdb_423e_b71d_ec7b0330fa50.slice/crio-33bfd9fe7cc021e8d82e48d92b344184ae519b0d7d7f22b2b804f4afb2acc1d5 WatchSource:0}: Error finding container 33bfd9fe7cc021e8d82e48d92b344184ae519b0d7d7f22b2b804f4afb2acc1d5: Status 404 returned error can't find the container with id 33bfd9fe7cc021e8d82e48d92b344184ae519b0d7d7f22b2b804f4afb2acc1d5
Nov 03 18:46:50.242690 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.242672    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"]
Nov 03 18:46:50.253838 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.253810    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92kft\" (UniqueName: \"kubernetes.io/projected/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-kube-api-access-92kft\") pod \"authorino-6576df7764-h75vj\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") " pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.253976 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.253858    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-tls-cert\") pod \"authorino-6576df7764-h75vj\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") " pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.342159 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:46:50.342112    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dae7546_f63e_49b3_9881_ee51a985942e.slice/crio-5a84881a9119325acac84fec0d0cb47bfc8a03c45b8ca7cd04a054807f1f3f7f WatchSource:0}: Error finding container 5a84881a9119325acac84fec0d0cb47bfc8a03c45b8ca7cd04a054807f1f3f7f: Status 404 returned error can't find the container with id 5a84881a9119325acac84fec0d0cb47bfc8a03c45b8ca7cd04a054807f1f3f7f
Nov 03 18:46:50.344345 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.344326    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68c48687b6-cszvm"]
Nov 03 18:46:50.354760 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.354739    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-tls-cert\") pod \"authorino-6576df7764-h75vj\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") " pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.354876 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.354812    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92kft\" (UniqueName: \"kubernetes.io/projected/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-kube-api-access-92kft\") pod \"authorino-6576df7764-h75vj\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") " pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.357256 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.357235    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-tls-cert\") pod \"authorino-6576df7764-h75vj\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") " pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.417821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.417792    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92kft\" (UniqueName: \"kubernetes.io/projected/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-kube-api-access-92kft\") pod \"authorino-6576df7764-h75vj\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") " pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.428604 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.428585    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:46:50.818409 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.818380    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c48687b6-cszvm" event={"ID":"5dae7546-f63e-49b3-9881-ee51a985942e","Type":"ContainerStarted","Data":"5a84881a9119325acac84fec0d0cb47bfc8a03c45b8ca7cd04a054807f1f3f7f"}
Nov 03 18:46:50.819782 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.819755    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4" event={"ID":"c7f9a639-2fdb-423e-b71d-ec7b0330fa50","Type":"ContainerStarted","Data":"33bfd9fe7cc021e8d82e48d92b344184ae519b0d7d7f22b2b804f4afb2acc1d5"}
Nov 03 18:46:50.819886 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.819767    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bfk8g"
Nov 03 18:46:50.821109 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:50.821088    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6576df7764-h75vj"]
Nov 03 18:46:51.054639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:51.054566    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bfk8g"]
Nov 03 18:46:51.060383 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:51.060352    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bfk8g"]
Nov 03 18:46:51.826863 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:51.826821    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6576df7764-h75vj" event={"ID":"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1","Type":"ContainerStarted","Data":"1c460415680c5f52e024e7cd4e51307951630933fa9b9b977366faaf5ed44607"}
Nov 03 18:46:52.995241 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:52.995206    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936e953f-9d94-4cf0-938d-0f3692825d79" path="/var/lib/kubelet/pods/936e953f-9d94-4cf0-938d-0f3692825d79/volumes"
Nov 03 18:46:54.854089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:54.854049    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4" event={"ID":"c7f9a639-2fdb-423e-b71d-ec7b0330fa50","Type":"ContainerStarted","Data":"be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e"}
Nov 03 18:46:54.854514 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:54.854192    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:46:54.975873 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:54.975808    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4" podStartSLOduration=2.358243619 podStartE2EDuration="5.975787936s" totalImagesPullingTime="3.617544317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:54.975623468 +0000 UTC m=+544.124318068" watchObservedRunningTime="2026-11-03 18:46:54.975787936 +0000 UTC m=+544.124482510"
Nov 03 18:46:55.859620 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:55.859531    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6576df7764-h75vj" event={"ID":"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1","Type":"ContainerStarted","Data":"e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771"}
Nov 03 18:46:55.861159 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:55.861113    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c48687b6-cszvm" event={"ID":"5dae7546-f63e-49b3-9881-ee51a985942e","Type":"ContainerStarted","Data":"5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd"}
Nov 03 18:46:55.861159 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:55.861129    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68c48687b6-cszvm" podUID="5dae7546-f63e-49b3-9881-ee51a985942e" containerName="authorino" containerID="cri-o://5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd" gracePeriod=30
Nov 03 18:46:55.979867 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:55.979813    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6576df7764-h75vj" podStartSLOduration=1.252012044 podStartE2EDuration="5.979797173s" totalImagesPullingTime="4.727785129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:55.979120832 +0000 UTC m=+545.127815403" watchObservedRunningTime="2026-11-03 18:46:55.979797173 +0000 UTC m=+545.128491789"
Nov 03 18:46:56.097782 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.097740    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68c48687b6-cszvm" podStartSLOduration=1.883337936 podStartE2EDuration="7.097725087s" totalImagesPullingTime="5.214387151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:56.097010288 +0000 UTC m=+545.245704860" watchObservedRunningTime="2026-11-03 18:46:56.097725087 +0000 UTC m=+545.246419658"
Nov 03 18:46:56.104218 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.104193    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c48687b6-cszvm"
Nov 03 18:46:56.105318 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.105297    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dae7546-f63e-49b3-9881-ee51a985942e-kube-api-access-gg2ld\" (UniqueName: \"kubernetes.io/projected/5dae7546-f63e-49b3-9881-ee51a985942e-kube-api-access-gg2ld\") pod \"5dae7546-f63e-49b3-9881-ee51a985942e\" (UID: \"5dae7546-f63e-49b3-9881-ee51a985942e\") "
Nov 03 18:46:56.107459 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.107439    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dae7546-f63e-49b3-9881-ee51a985942e-kube-api-access-gg2ld" pod "5dae7546-f63e-49b3-9881-ee51a985942e" (UID: "5dae7546-f63e-49b3-9881-ee51a985942e"). InnerVolumeSpecName "kube-api-access-gg2ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:46:56.206532 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.206500    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2ld\" (UniqueName: \"kubernetes.io/projected/5dae7546-f63e-49b3-9881-ee51a985942e-kube-api-access-gg2ld\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:46:56.865647 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.865561    4489 generic.go:356] "Generic (PLEG): container finished" podID="5dae7546-f63e-49b3-9881-ee51a985942e" containerID="5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd" exitCode=0
Nov 03 18:46:56.865647 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.865607    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68c48687b6-cszvm"
Nov 03 18:46:56.866172 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.865656    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c48687b6-cszvm" event={"ID":"5dae7546-f63e-49b3-9881-ee51a985942e","Type":"ContainerDied","Data":"5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd"}
Nov 03 18:46:56.866172 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.865703    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68c48687b6-cszvm" event={"ID":"5dae7546-f63e-49b3-9881-ee51a985942e","Type":"ContainerDied","Data":"5a84881a9119325acac84fec0d0cb47bfc8a03c45b8ca7cd04a054807f1f3f7f"}
Nov 03 18:46:56.866172 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.865725    4489 scope.go:122] "RemoveContainer" containerID="5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd"
Nov 03 18:46:56.873809 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.873790    4489 scope.go:122] "RemoveContainer" containerID="5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd"
Nov 03 18:46:56.874110 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:46:56.874091    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd\": container with ID starting with 5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd not found: ID does not exist" containerID="5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd"
Nov 03 18:46:56.874184 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:56.874118    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd"} err="failed to get container status \"5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd\": rpc error: code = NotFound desc = could not find container \"5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd\": container with ID starting with 5457ee9a9a8ec0d6f48aafe904a65b32e4dfb6dbd01531f98cb5807f88f531fd not found: ID does not exist"
Nov 03 18:46:57.035409 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:57.035374    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68c48687b6-cszvm"]
Nov 03 18:46:57.038960 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:57.038935    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68c48687b6-cszvm"]
Nov 03 18:46:58.993290 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:46:58.993259    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dae7546-f63e-49b3-9881-ee51a985942e" path="/var/lib/kubelet/pods/5dae7546-f63e-49b3-9881-ee51a985942e/volumes"
Nov 03 18:47:00.866694 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:47:00.866669    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 18:47:17.528325 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:47:17.528289    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:47:47.529351 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:47:47.529314    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:48:17.529203 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:48:17.529166    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:48:47.529178 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:48:47.529117    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:48:53.834658 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.834624    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"]
Nov 03 18:48:53.835131 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.834895    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5dae7546-f63e-49b3-9881-ee51a985942e" containerName="authorino"
Nov 03 18:48:53.835131 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.834905    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dae7546-f63e-49b3-9881-ee51a985942e" containerName="authorino"
Nov 03 18:48:53.835131 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.834968    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dae7546-f63e-49b3-9881-ee51a985942e" containerName="authorino"
Nov 03 18:48:53.837615 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.837598    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.866033 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.865998    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91928076-5652-40ca-ba0e-7cfe3ead9cb6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.866033 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.866030    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.866242 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.866052    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.866242 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.866183    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.866242 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.866232    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gsrh\" (UniqueName: \"kubernetes.io/projected/91928076-5652-40ca-ba0e-7cfe3ead9cb6-kube-api-access-7gsrh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.866358 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.866316    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.946661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.946630    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xxl7k\""
Nov 03 18:48:53.946861 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.946746    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 18:48:53.946921 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.946886    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 18:48:53.946956 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.946919    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 18:48:53.947794 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.947775    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"]
Nov 03 18:48:53.966968 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.966934    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsrh\" (UniqueName: \"kubernetes.io/projected/91928076-5652-40ca-ba0e-7cfe3ead9cb6-kube-api-access-7gsrh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.966968 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.966971    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.967240 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.966994    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91928076-5652-40ca-ba0e-7cfe3ead9cb6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.967240 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.967020    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.967240 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.967047    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.967240 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.967088    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.967465 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.967430    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.967503 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.967461    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.967541 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.967519    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.969572 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.969548    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91928076-5652-40ca-ba0e-7cfe3ead9cb6-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:53.969721 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:53.969704    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91928076-5652-40ca-ba0e-7cfe3ead9cb6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:54.032096 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:54.032066    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gsrh\" (UniqueName: \"kubernetes.io/projected/91928076-5652-40ca-ba0e-7cfe3ead9cb6-kube-api-access-7gsrh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rh6nq\" (UID: \"91928076-5652-40ca-ba0e-7cfe3ead9cb6\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:54.149710 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:54.149625    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:54.521544 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:48:54.521514    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91928076_5652_40ca_ba0e_7cfe3ead9cb6.slice/crio-10b4e9b473e300063f0710f6077e75d9a1c7b5aefe8170091ba732d15f9247d0 WatchSource:0}: Error finding container 10b4e9b473e300063f0710f6077e75d9a1c7b5aefe8170091ba732d15f9247d0: Status 404 returned error can't find the container with id 10b4e9b473e300063f0710f6077e75d9a1c7b5aefe8170091ba732d15f9247d0
Nov 03 18:48:54.522905 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:54.522882    4489 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:48:54.524089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:54.524066    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"]
Nov 03 18:48:55.308820 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:55.308776    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq" event={"ID":"91928076-5652-40ca-ba0e-7cfe3ead9cb6","Type":"ContainerStarted","Data":"10b4e9b473e300063f0710f6077e75d9a1c7b5aefe8170091ba732d15f9247d0"}
Nov 03 18:48:57.317414 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:57.317372    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq" event={"ID":"91928076-5652-40ca-ba0e-7cfe3ead9cb6","Type":"ContainerStarted","Data":"467ee2674ca80710354172700ada68bcd660dbcb4982439b14baefd9931e7142"}
Nov 03 18:48:57.443411 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:57.443360    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq" podStartSLOduration=2.207989654 podStartE2EDuration="4.443346276s" totalImagesPullingTime="2.235356622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:57.442476153 +0000 UTC m=+666.591170724" watchObservedRunningTime="2026-11-03 18:48:57.443346276 +0000 UTC m=+666.592040846"
Nov 03 18:48:58.150662 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.150624    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:48:58.268637 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.268604    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"]
Nov 03 18:48:58.271887 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.271868    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.302389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.302358    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.302389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.302393    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.302642 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.302487    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4216e6-0db2-41a5-8fdd-91320e6681ea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.302642 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.302523    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jb5f\" (UniqueName: \"kubernetes.io/projected/cf4216e6-0db2-41a5-8fdd-91320e6681ea-kube-api-access-4jb5f\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.302642 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.302624    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.302796 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.302663    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.380313 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.380273    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 18:48:58.381475 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.381455    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"]
Nov 03 18:48:58.403728 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.403671    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4216e6-0db2-41a5-8fdd-91320e6681ea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.403728 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.403713    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jb5f\" (UniqueName: \"kubernetes.io/projected/cf4216e6-0db2-41a5-8fdd-91320e6681ea-kube-api-access-4jb5f\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.403835 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.403738    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.403920 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.403894    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.403985 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.403965    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.404057 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.404031    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.404247 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.404224    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.404297 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.404277    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.404352 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.404321    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.406229 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.406203    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4216e6-0db2-41a5-8fdd-91320e6681ea-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.406459 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.406438    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4216e6-0db2-41a5-8fdd-91320e6681ea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.470242 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.470209    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jb5f\" (UniqueName: \"kubernetes.io/projected/cf4216e6-0db2-41a5-8fdd-91320e6681ea-kube-api-access-4jb5f\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9\" (UID: \"cf4216e6-0db2-41a5-8fdd-91320e6681ea\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.588015 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.587982    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:48:58.956596 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:48:58.956553    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4216e6_0db2_41a5_8fdd_91320e6681ea.slice/crio-dc9bd4a2c61f4ad72d8b07169e6076bbcb6c8246998a8bc70877720f7ea36aa4 WatchSource:0}: Error finding container dc9bd4a2c61f4ad72d8b07169e6076bbcb6c8246998a8bc70877720f7ea36aa4: Status 404 returned error can't find the container with id dc9bd4a2c61f4ad72d8b07169e6076bbcb6c8246998a8bc70877720f7ea36aa4
Nov 03 18:48:58.958205 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:58.958177    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"]
Nov 03 18:48:59.329370 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:48:59.329286    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9" event={"ID":"cf4216e6-0db2-41a5-8fdd-91320e6681ea","Type":"ContainerStarted","Data":"dc9bd4a2c61f4ad72d8b07169e6076bbcb6c8246998a8bc70877720f7ea36aa4"}
Nov 03 18:49:00.334196 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:00.334138    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9" event={"ID":"cf4216e6-0db2-41a5-8fdd-91320e6681ea","Type":"ContainerStarted","Data":"cd87808056117fed1aee771c4846e7d3b3128507e68b27949ecc364f48a5f448"}
Nov 03 18:49:00.457102 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:00.457049    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9" podStartSLOduration=1.896209685 podStartE2EDuration="2.45703386s" totalImagesPullingTime="560.824175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:00.455916712 +0000 UTC m=+669.604611283" watchObservedRunningTime="2026-11-03 18:49:00.45703386 +0000 UTC m=+669.605728431"
Nov 03 18:49:00.588889 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:00.588805    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:49:02.194089 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.194006    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"]
Nov 03 18:49:02.197449 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.197430    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.235837 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.235798    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb6520e-6a2d-43c6-91d8-a19e784b3180-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.236017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.235852    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.236017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.236007    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.236123 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.236035    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.236123 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.236073    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.236211 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.236169    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9vtq\" (UniqueName: \"kubernetes.io/projected/0cb6520e-6a2d-43c6-91d8-a19e784b3180-kube-api-access-h9vtq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.253792 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.253747    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"]
Nov 03 18:49:02.303600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.303566    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 18:49:02.336961 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.336915    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.336961 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.336963    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.337222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.336995    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.337222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.337019    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9vtq\" (UniqueName: \"kubernetes.io/projected/0cb6520e-6a2d-43c6-91d8-a19e784b3180-kube-api-access-h9vtq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.337222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.337063    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb6520e-6a2d-43c6-91d8-a19e784b3180-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.337222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.337106    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.337446 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.337424    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.337446 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.337439    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.337551 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.337453    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.339544 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.339523    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb6520e-6a2d-43c6-91d8-a19e784b3180-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.339722 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.339706    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb6520e-6a2d-43c6-91d8-a19e784b3180-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.400914 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.400875    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9vtq\" (UniqueName: \"kubernetes.io/projected/0cb6520e-6a2d-43c6-91d8-a19e784b3180-kube-api-access-h9vtq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k44p2\" (UID: \"0cb6520e-6a2d-43c6-91d8-a19e784b3180\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.510344 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.510258    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:02.891937 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:49:02.891900    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb6520e_6a2d_43c6_91d8_a19e784b3180.slice/crio-64d80278f2b90c1c6d06a5e9edbaec2c827dc232c88cfbac382091c566c1db8b WatchSource:0}: Error finding container 64d80278f2b90c1c6d06a5e9edbaec2c827dc232c88cfbac382091c566c1db8b: Status 404 returned error can't find the container with id 64d80278f2b90c1c6d06a5e9edbaec2c827dc232c88cfbac382091c566c1db8b
Nov 03 18:49:02.894466 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:02.894445    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"]
Nov 03 18:49:03.347950 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:03.347918    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2" event={"ID":"0cb6520e-6a2d-43c6-91d8-a19e784b3180","Type":"ContainerStarted","Data":"64d80278f2b90c1c6d06a5e9edbaec2c827dc232c88cfbac382091c566c1db8b"}
Nov 03 18:49:04.150766 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:04.150723    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:49:04.163836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:04.163808    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:49:04.354724 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:04.354680    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2" event={"ID":"0cb6520e-6a2d-43c6-91d8-a19e784b3180","Type":"ContainerStarted","Data":"136f1160feb2b0bbc610ec3a9160449085ded5535fc6567915361e026684cad3"}
Nov 03 18:49:04.366128 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:04.366103    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rh6nq"
Nov 03 18:49:04.477009 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:04.476950    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2" podStartSLOduration=1.888693531 podStartE2EDuration="2.476934388s" totalImagesPullingTime="588.240857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:04.476280091 +0000 UTC m=+673.624974666" watchObservedRunningTime="2026-11-03 18:49:04.476934388 +0000 UTC m=+673.625628959"
Nov 03 18:49:04.510549 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:04.510498    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:08.589216 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:08.589180    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:49:08.601702 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:08.601674    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:49:09.389501 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.389470    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9"
Nov 03 18:49:09.825707 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.825672    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"]
Nov 03 18:49:09.833076 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.833052    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.890180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.887730    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"]
Nov 03 18:49:09.894912 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.894883    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.894936    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.894966    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.894988    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.895012    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.895044    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895354 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.895160    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895354 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.895180    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.895354 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.895198    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbzvt\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-kube-api-access-dbzvt\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.938131 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.938094    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 18:49:09.938300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.938283    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h6x4t\""
Nov 03 18:49:09.996421 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996383    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996421 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996422    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996441    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbzvt\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-kube-api-access-dbzvt\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996472    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996501    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996527    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996552    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996567    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996583    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996950 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996896    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.996950 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.996937    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.997061 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.997044    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.997115 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.997082    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.997273 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.997251    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.999003 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.998973    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:09.999254 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:09.999238    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:10.059905 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:10.059864    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:10.060058 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:10.059935    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbzvt\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-kube-api-access-dbzvt\") pod \"maas-default-gateway-openshift-default-7854949c9-dffvr\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:10.148347 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:10.148265    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:10.568521 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:49:10.568492    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde364ef_3c12_4f53_ad45_a1512ef63c7d.slice/crio-71fc96347e4fbc8efa916df3c6b904e6d92db404deaea2f8dbf186f1e89f2668 WatchSource:0}: Error finding container 71fc96347e4fbc8efa916df3c6b904e6d92db404deaea2f8dbf186f1e89f2668: Status 404 returned error can't find the container with id 71fc96347e4fbc8efa916df3c6b904e6d92db404deaea2f8dbf186f1e89f2668
Nov 03 18:49:10.571116 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:10.571092    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"]
Nov 03 18:49:11.388207 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:11.388166    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" event={"ID":"bde364ef-3c12-4f53-ad45-a1512ef63c7d","Type":"ContainerStarted","Data":"71fc96347e4fbc8efa916df3c6b904e6d92db404deaea2f8dbf186f1e89f2668"}
Nov 03 18:49:12.510859 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:12.510814    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:12.523646 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:12.523620    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:13.413842 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:13.413811    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k44p2"
Nov 03 18:49:14.739783 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:14.739755    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:49:14.740031 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:14.739822    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:49:14.740031 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:14.739852    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:49:15.407489 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:15.407451    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" event={"ID":"bde364ef-3c12-4f53-ad45-a1512ef63c7d","Type":"ContainerStarted","Data":"c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd"}
Nov 03 18:49:15.596580 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:15.596521    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podStartSLOduration=2.4281197629999998 podStartE2EDuration="6.596501311s" totalImagesPullingTime="4.168381548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:15.595899036 +0000 UTC m=+684.744593606" watchObservedRunningTime="2026-11-03 18:49:15.596501311 +0000 UTC m=+684.745195883"
Nov 03 18:49:16.161998 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:16.161964    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:16.162653 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:16.162626    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-dffvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:49:16.162843 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:16.162817    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 18:49:17.168169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:17.160535    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-dffvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:49:17.168169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:17.160598    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 18:49:17.532650 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:49:17.532574    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:49:18.169172 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.159698    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-dffvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:49:18.169172 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.159782    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 18:49:18.297495 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.297465    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hjb4p"]
Nov 03 18:49:18.301973 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.301956    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.360199 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.360165    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hjb4p"]
Nov 03 18:49:18.388024 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.387988    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.388024 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.388025    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jntw\" (UniqueName: \"kubernetes.io/projected/edeeb111-376a-40b7-85de-3d95677ad5d4-kube-api-access-9jntw\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.388259 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.388049    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-home\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.388259 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.388173    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.388259 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.388217    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.388358 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.388274    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edeeb111-376a-40b7-85de-3d95677ad5d4-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.409872 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.409837    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 18:49:18.489307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489217    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489256    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jntw\" (UniqueName: \"kubernetes.io/projected/edeeb111-376a-40b7-85de-3d95677ad5d4-kube-api-access-9jntw\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489283    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-home\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489308    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489334    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489382    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edeeb111-376a-40b7-85de-3d95677ad5d4-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489807 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489741    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-home\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489807 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489781    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.489890 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.489828    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.491693 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.491667    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edeeb111-376a-40b7-85de-3d95677ad5d4-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.492071 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.492050    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edeeb111-376a-40b7-85de-3d95677ad5d4-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.576172 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.576108    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jntw\" (UniqueName: \"kubernetes.io/projected/edeeb111-376a-40b7-85de-3d95677ad5d4-kube-api-access-9jntw\") pod \"echo-simulator-kserve-64b6f8f96c-hjb4p\" (UID: \"edeeb111-376a-40b7-85de-3d95677ad5d4\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:18.615181 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:18.615132    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:19.002746 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:49:19.002715    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedeeb111_376a_40b7_85de_3d95677ad5d4.slice/crio-79c151739b7dc87966224526fa3ce93ca89254d3ecc8414cbbca157db26f6c6a WatchSource:0}: Error finding container 79c151739b7dc87966224526fa3ce93ca89254d3ecc8414cbbca157db26f6c6a: Status 404 returned error can't find the container with id 79c151739b7dc87966224526fa3ce93ca89254d3ecc8414cbbca157db26f6c6a
Nov 03 18:49:19.008050 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:19.008031    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hjb4p"]
Nov 03 18:49:19.169165 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:19.168555    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-dffvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:49:19.169165 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:19.168615    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 18:49:19.424327 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:19.424286    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p" event={"ID":"edeeb111-376a-40b7-85de-3d95677ad5d4","Type":"ContainerStarted","Data":"79c151739b7dc87966224526fa3ce93ca89254d3ecc8414cbbca157db26f6c6a"}
Nov 03 18:49:20.173170 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:20.166567    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-dffvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:49:20.173170 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:20.166629    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 18:49:21.149505 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.149462    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-dffvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:49:21.149693 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.149534    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 18:49:21.437897 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.437844    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p" event={"ID":"edeeb111-376a-40b7-85de-3d95677ad5d4","Type":"ContainerStarted","Data":"4bafbab981ff2a4a3193e897ced84642f16a9a020c7e57c3534e7d45592e69c6"}
Nov 03 18:49:21.560419 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.560369    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p" podStartSLOduration=1.261221209 podStartE2EDuration="3.56035468s" totalImagesPullingTime="2.299133471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:21.559493639 +0000 UTC m=+690.708188245" watchObservedRunningTime="2026-11-03 18:49:21.56035468 +0000 UTC m=+690.709049251"
Nov 03 18:49:21.615749 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.615687    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:21.879102 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.879012    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"]
Nov 03 18:49:21.882436 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.882414    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:21.921278 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.921228    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:21.921278 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.921280    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:21.921498 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.921366    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:21.921498 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.921479    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96ng\" (UniqueName: \"kubernetes.io/projected/a1a211ed-737a-4f7b-bc35-fd1021ac8098-kube-api-access-z96ng\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:21.921573 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.921524    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a211ed-737a-4f7b-bc35-fd1021ac8098-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:21.921616 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.921576    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:21.938283 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.938250    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 18:49:21.990882 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:21.990852    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"]
Nov 03 18:49:22.022113 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022075    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022333 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022124    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96ng\" (UniqueName: \"kubernetes.io/projected/a1a211ed-737a-4f7b-bc35-fd1021ac8098-kube-api-access-z96ng\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022333 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022164    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a211ed-737a-4f7b-bc35-fd1021ac8098-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022333 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022193    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022333 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022244    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022558 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022344    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022609 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022571    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022722 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022691    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.022958 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.022930    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.024981 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.024958    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1a211ed-737a-4f7b-bc35-fd1021ac8098-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.025087 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.025042    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1a211ed-737a-4f7b-bc35-fd1021ac8098-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.085602 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.085575    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96ng\" (UniqueName: \"kubernetes.io/projected/a1a211ed-737a-4f7b-bc35-fd1021ac8098-kube-api-access-z96ng\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf\" (UID: \"a1a211ed-737a-4f7b-bc35-fd1021ac8098\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.199195 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.199137    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:22.571188 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:49:22.571155    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a211ed_737a_4f7b_bc35_fd1021ac8098.slice/crio-86368b6bdfa5b952099f25761be06f6045bd9a9a6161a275c36a5ab5f167aa81 WatchSource:0}: Error finding container 86368b6bdfa5b952099f25761be06f6045bd9a9a6161a275c36a5ab5f167aa81: Status 404 returned error can't find the container with id 86368b6bdfa5b952099f25761be06f6045bd9a9a6161a275c36a5ab5f167aa81
Nov 03 18:49:22.574217 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:22.574192    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"]
Nov 03 18:49:23.191375 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.191339    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-dffvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:49:23.191495 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.191387    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:49:23.407782 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.407748    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:23.408263 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.408245    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:23.409000 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.408984    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:49:23.446678 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.446648    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf" event={"ID":"a1a211ed-737a-4f7b-bc35-fd1021ac8098","Type":"ContainerStarted","Data":"0a280706663d19528ccd52ff27468df7ea8c70493f371effdf65a9244a564ee7"}
Nov 03 18:49:23.446851 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.446690    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf" event={"ID":"a1a211ed-737a-4f7b-bc35-fd1021ac8098","Type":"ContainerStarted","Data":"86368b6bdfa5b952099f25761be06f6045bd9a9a6161a275c36a5ab5f167aa81"}
Nov 03 18:49:23.830920 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:23.830831    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf" podStartSLOduration=2.279134159 podStartE2EDuration="2.83081572s" totalImagesPullingTime="551.681561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:23.829771532 +0000 UTC m=+692.978466126" watchObservedRunningTime="2026-11-03 18:49:23.83081572 +0000 UTC m=+692.979510294"
Nov 03 18:49:24.199405 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:24.199372    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:25.476774 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.476738    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"]
Nov 03 18:49:25.480192 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.480171    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.538829 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.538796    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"]
Nov 03 18:49:25.558212 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.558131    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.558375 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.558280    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.558444 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.558421    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.558507 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.558476    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.558562 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.558511    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp95s\" (UniqueName: \"kubernetes.io/projected/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-kube-api-access-fp95s\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.558562 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.558542    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.587132 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.587095    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 18:49:25.659755 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.659715    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.659940 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.659778    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.659940 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.659838    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.659940 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.659865    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.659940 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.659886    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp95s\" (UniqueName: \"kubernetes.io/projected/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-kube-api-access-fp95s\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.659940 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.659922    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.660302 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.660278    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.660366 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.660333    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.660534 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.660480    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.662866 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.662837    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.663112 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.663091    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.727197 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.727106    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp95s\" (UniqueName: \"kubernetes.io/projected/e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6-kube-api-access-fp95s\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v\" (UID: \"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:25.792536 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:25.792473    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:26.185405 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:26.185220    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"]
Nov 03 18:49:26.463058 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:26.462953    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" event={"ID":"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6","Type":"ContainerStarted","Data":"28a2f44daec1132e1e1d885cdaa8d0867d4d73e007b39d244db5831fc909e50b"}
Nov 03 18:49:27.468395 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:27.468356    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" event={"ID":"e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6","Type":"ContainerStarted","Data":"b4269de3bddb86c6e418a88d95b2a297a94503d4727d7dbe3f741f90ed18e4df"}
Nov 03 18:49:27.591742 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:27.591692    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podStartSLOduration=1.993245531 podStartE2EDuration="2.591678024s" totalImagesPullingTime="598.432493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:27.590471232 +0000 UTC m=+696.739165813" watchObservedRunningTime="2026-11-03 18:49:27.591678024 +0000 UTC m=+696.740372595"
Nov 03 18:49:27.792832 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:27.792753    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:28.616201 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:28.616161    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:28.628836 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:28.628808    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:29.487686 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:29.487650    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hjb4p"
Nov 03 18:49:32.200327 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:32.200289    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:32.212519 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:32.212496    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:32.498832 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:32.498753    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf"
Nov 03 18:49:35.793629 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:35.793596    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:35.806029 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:35.806002    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:36.518367 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:36.518339    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 18:49:40.092086 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.092052    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"]
Nov 03 18:49:40.095347 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.095331    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.182532 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.182502    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"]
Nov 03 18:49:40.187927 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.187899    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.187952    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.188030    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188198 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.188097    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188239 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.188206    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188239 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.188228    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188336 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.188262    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188336 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.188304    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.188418 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.188342    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwpm\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-kube-api-access-9mwpm\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289251 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289194    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289459 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289271    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289459 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289295    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289459 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289316    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwpm\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-kube-api-access-9mwpm\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289459 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289346    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289475    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289538    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289601    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.289731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289659    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.290004 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289744    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.290004 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.289889    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.290117 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.290039    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.290254 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.290229    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.290483 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.290462    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.291804 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.291783    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.292177 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.292136    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.353084 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.353014    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwpm\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-kube-api-access-9mwpm\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.353084 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.353011    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-bw6rx\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.406950 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.406908    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:40.823617 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:49:40.823577    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab38159d_89dc_48a9_a4e7_9333e3436dcd.slice/crio-e2e8b57c662b98a332b442b695481583b19927a30de039cb578ff12b278c8cfc WatchSource:0}: Error finding container e2e8b57c662b98a332b442b695481583b19927a30de039cb578ff12b278c8cfc: Status 404 returned error can't find the container with id e2e8b57c662b98a332b442b695481583b19927a30de039cb578ff12b278c8cfc
Nov 03 18:49:40.825991 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.825958    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:49:40.826097 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.826026    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:49:40.826097 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.826059    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:49:40.826713 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:40.826691    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"]
Nov 03 18:49:41.528346 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:41.528302    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" event={"ID":"ab38159d-89dc-48a9-a4e7-9333e3436dcd","Type":"ContainerStarted","Data":"d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0"}
Nov 03 18:49:41.528847 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:41.528357    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" event={"ID":"ab38159d-89dc-48a9-a4e7-9333e3436dcd","Type":"ContainerStarted","Data":"e2e8b57c662b98a332b442b695481583b19927a30de039cb578ff12b278c8cfc"}
Nov 03 18:49:41.773169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:41.767599    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podStartSLOduration=1.767565461 podStartE2EDuration="1.767565461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:41.705003167 +0000 UTC m=+710.853697739" watchObservedRunningTime="2026-11-03 18:49:41.767565461 +0000 UTC m=+710.916260036"
Nov 03 18:49:42.407209 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:42.407171    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:42.408557 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:42.408532    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-bw6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:49:42.408678 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:42.408582    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 18:49:43.407684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:43.407650    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-bw6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:49:43.408084 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:43.407701    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 18:49:44.407572 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:44.407536    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-bw6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:49:44.407719 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:44.407597    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 18:49:45.407527 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:45.407491    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-bw6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:49:45.407711 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:45.407545    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 18:49:46.407570 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:46.407536    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-bw6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:49:46.407970 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:46.407601    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 18:49:47.407191 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:47.407136    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-bw6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:49:47.407345 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:47.407249    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 18:49:47.529494 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:49:47.529451    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:49:49.013670 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:49.013635    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:49.014170 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:49.013984    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:49.014930 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:49.014911    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:49:55.233049 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.233015    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-757fd8969f-t8vmb"]
Nov 03 18:49:55.243475 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.243441    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.290443 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.290414    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757fd8969f-t8vmb"]
Nov 03 18:49:55.328600 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.328571    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5-tls-cert\") pod \"authorino-757fd8969f-t8vmb\" (UID: \"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5\") " pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.328751 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.328605    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87sks\" (UniqueName: \"kubernetes.io/projected/dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5-kube-api-access-87sks\") pod \"authorino-757fd8969f-t8vmb\" (UID: \"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5\") " pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.430090 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.430047    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5-tls-cert\") pod \"authorino-757fd8969f-t8vmb\" (UID: \"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5\") " pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.430090 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.430087    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sks\" (UniqueName: \"kubernetes.io/projected/dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5-kube-api-access-87sks\") pod \"authorino-757fd8969f-t8vmb\" (UID: \"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5\") " pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.432705 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.432677    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5-tls-cert\") pod \"authorino-757fd8969f-t8vmb\" (UID: \"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5\") " pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.494300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.494220    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sks\" (UniqueName: \"kubernetes.io/projected/dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5-kube-api-access-87sks\") pod \"authorino-757fd8969f-t8vmb\" (UID: \"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5\") " pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.555384 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.555349    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757fd8969f-t8vmb"
Nov 03 18:49:55.925853 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:49:55.925815    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddde2f2d2_a1c6_4b6a_a542_a4e655cd04c5.slice/crio-72b7130f9992f8ddb3ba52375dbd75aaeebce4927199cdbef21a29bd6ad04ba6 WatchSource:0}: Error finding container 72b7130f9992f8ddb3ba52375dbd75aaeebce4927199cdbef21a29bd6ad04ba6: Status 404 returned error can't find the container with id 72b7130f9992f8ddb3ba52375dbd75aaeebce4927199cdbef21a29bd6ad04ba6
Nov 03 18:49:55.929029 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:55.929010    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757fd8969f-t8vmb"]
Nov 03 18:49:56.592634 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:49:56.592598    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757fd8969f-t8vmb" event={"ID":"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5","Type":"ContainerStarted","Data":"72b7130f9992f8ddb3ba52375dbd75aaeebce4927199cdbef21a29bd6ad04ba6"}
Nov 03 18:50:08.644356 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:08.644296    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757fd8969f-t8vmb" event={"ID":"dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5","Type":"ContainerStarted","Data":"3028367bb655864ab780d9087bc6b345988ce06878579768f8733fab7ce4cad2"}
Nov 03 18:50:08.844590 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:08.844521    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-757fd8969f-t8vmb" podStartSLOduration=2.16926961 podStartE2EDuration="13.844498233s" totalImagesPullingTime="11.675228623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:08.796252264 +0000 UTC m=+737.944946851" watchObservedRunningTime="2026-11-03 18:50:08.844498233 +0000 UTC m=+737.993192879"
Nov 03 18:50:08.847208 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:08.847174    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6576df7764-h75vj"]
Nov 03 18:50:08.847426 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:08.847399    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6576df7764-h75vj" podUID="b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" containerName="authorino" containerID="cri-o://e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771" gracePeriod=30
Nov 03 18:50:09.123034 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.123001    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:50:09.163323 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.163290    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-tls-cert\" (UniqueName: \"kubernetes.io/secret/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-tls-cert\") pod \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") "
Nov 03 18:50:09.163529 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.163367    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-kube-api-access-92kft\" (UniqueName: \"kubernetes.io/projected/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-kube-api-access-92kft\") pod \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\" (UID: \"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1\") "
Nov 03 18:50:09.165631 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.165600    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-kube-api-access-92kft" pod "b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" (UID: "b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1"). InnerVolumeSpecName "kube-api-access-92kft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:50:09.174262 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.174227    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-tls-cert" pod "b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" (UID: "b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:50:09.264684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.264640    4489 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-tls-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:50:09.264684 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.264675    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92kft\" (UniqueName: \"kubernetes.io/projected/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1-kube-api-access-92kft\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:50:09.650391 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.650298    4489 generic.go:356] "Generic (PLEG): container finished" podID="b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" containerID="e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771" exitCode=0
Nov 03 18:50:09.650391 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.650352    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6576df7764-h75vj"
Nov 03 18:50:09.650929 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.650388    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6576df7764-h75vj" event={"ID":"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1","Type":"ContainerDied","Data":"e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771"}
Nov 03 18:50:09.650929 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.650439    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6576df7764-h75vj" event={"ID":"b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1","Type":"ContainerDied","Data":"1c460415680c5f52e024e7cd4e51307951630933fa9b9b977366faaf5ed44607"}
Nov 03 18:50:09.650929 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.650461    4489 scope.go:122] "RemoveContainer" containerID="e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771"
Nov 03 18:50:09.659330 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.659313    4489 scope.go:122] "RemoveContainer" containerID="e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771"
Nov 03 18:50:09.659631 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:50:09.659606    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771\": container with ID starting with e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771 not found: ID does not exist" containerID="e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771"
Nov 03 18:50:09.659677 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.659642    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771"} err="failed to get container status \"e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771\": rpc error: code = NotFound desc = could not find container \"e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771\": container with ID starting with e000669c2727d0187bfc77f27ce4361b5e6d92676e9f638bdfd672972e4ae771 not found: ID does not exist"
Nov 03 18:50:09.823624 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.823587    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6576df7764-h75vj"]
Nov 03 18:50:09.827867 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:09.827843    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6576df7764-h75vj"]
Nov 03 18:50:10.993330 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:10.993293    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" path="/var/lib/kubelet/pods/b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1/volumes"
Nov 03 18:50:17.530043 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:50:17.530003    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:50:47.529581 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:50:47.529547    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:50:59.228273 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.228238    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"]
Nov 03 18:50:59.228661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.228584    4489 cpu_manager.go:414] "containerMap: removing container" podUID="b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" containerName="authorino"
Nov 03 18:50:59.228661 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.228594    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" containerName="authorino"
Nov 03 18:50:59.228734 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.228669    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81fa3ae-30c2-4ad6-ac67-25b8667b0ab1" containerName="authorino"
Nov 03 18:50:59.232966 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.232940    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.288067 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.288030    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a4cacd-openshift-default-dockercfg-hkxvc\""
Nov 03 18:50:59.321858 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.321813    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.321858 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.321863    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01570eb7-f2dd-42fc-8b61-35555c4c855e-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.322107 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.321884    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.322107 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.321951    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.322107 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.322014    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.322107 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.322040    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.322329 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.322160    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.322329 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.322202    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.322329 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.322218    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxmc\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-kube-api-access-zbxmc\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.341774 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.341745    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"]
Nov 03 18:50:59.422986 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.422951    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.422986 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.422991    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423269 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423057    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423269 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423103    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423269 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423129    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxmc\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-kube-api-access-zbxmc\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423269 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423207    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423269 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423246    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01570eb7-f2dd-42fc-8b61-35555c4c855e-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423524 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423275    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423524 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423317    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423636 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423591    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423693 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423657    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423745 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423696    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.423847 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423829    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.424033 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.423944    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01570eb7-f2dd-42fc-8b61-35555c4c855e-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.425540 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.425523    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.425638 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.425618    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.488989 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.488901    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.489166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.489031    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxmc\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-kube-api-access-zbxmc\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.531941 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.531910    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"]
Nov 03 18:50:59.536535 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.536514    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.545011 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.544988    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:50:59.598347 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.598310    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-355e1a-openshift-default-dockercfg-7nkhf\""
Nov 03 18:50:59.598699 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.598676    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"]
Nov 03 18:50:59.625830 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.625805    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626065 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626051    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626186 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626172    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626274    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626392 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626373    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw84c\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-kube-api-access-dw84c\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626468 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626407    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626537 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626505    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626628 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626611    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.626687 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.626672    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.727970 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.727938    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.727970 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.727975    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.727991    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728007    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw84c\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-kube-api-access-dw84c\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728024    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728058    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728085    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728119    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728287 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728185    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728611 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728437    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728611 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728518    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728611 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728584    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728742 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728588    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.728957 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.728939    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.730567 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.730550    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.730662 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.730642    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.791602 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.791531    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.791732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.791658    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw84c\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-kube-api-access-dw84c\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.849061 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.849033    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:50:59.967898 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:50:59.967856    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01570eb7_f2dd_42fc_8b61_35555c4c855e.slice/crio-22c8aaf87d016545becdfce965ac29f50d7086acfb2da165336a88ccb0ebb5e0 WatchSource:0}: Error finding container 22c8aaf87d016545becdfce965ac29f50d7086acfb2da165336a88ccb0ebb5e0: Status 404 returned error can't find the container with id 22c8aaf87d016545becdfce965ac29f50d7086acfb2da165336a88ccb0ebb5e0
Nov 03 18:50:59.969777 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.969754    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"]
Nov 03 18:50:59.970127 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.970093    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:50:59.970248 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.970205    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:50:59.970313 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:50:59.970255    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:00.239677 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:00.239640    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7e4ba8_cc0a_427b_aea8_cb387ad31a22.slice/crio-7bcdfa4c82734c85daf26bdaa9572659a924dfc86c76ccf957dd7f48ab285a40 WatchSource:0}: Error finding container 7bcdfa4c82734c85daf26bdaa9572659a924dfc86c76ccf957dd7f48ab285a40: Status 404 returned error can't find the container with id 7bcdfa4c82734c85daf26bdaa9572659a924dfc86c76ccf957dd7f48ab285a40
Nov 03 18:51:00.241819 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.241792    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"]
Nov 03 18:51:00.242301 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.242273    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:00.242365 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.242353    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:00.242403 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.242385    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:00.466525 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.466488    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"]
Nov 03 18:51:00.470009 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.469991    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.526685 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.526533    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3a7056-openshift-default-dockercfg-wjcwp\""
Nov 03 18:51:00.528947 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.528919    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"]
Nov 03 18:51:00.534889 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.534865    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535019 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.534899    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535019 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.534925    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535101 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.535031    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e4111b-5349-4747-b39d-adba3dd53310-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535101 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.535060    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535194 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.535097    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535261 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.535246    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e4111b-5349-4747-b39d-adba3dd53310-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.535276    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.535393 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.535368    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42mk\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-kube-api-access-m42mk\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636747 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636714    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636747 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636750    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636775    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636810    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e4111b-5349-4747-b39d-adba3dd53310-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636853    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636904    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636939    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e4111b-5349-4747-b39d-adba3dd53310-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.636983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.636969    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.637330 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.637002    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42mk\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-kube-api-access-m42mk\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.637330 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.637239    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.637430 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.637336    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.637430 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.637364    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.637527 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.637502    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.637738 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.637634    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e4111b-5349-4747-b39d-adba3dd53310-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.639392 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.639370    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.639495 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.639377    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e4111b-5349-4747-b39d-adba3dd53310-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.701254 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.701222    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42mk\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-kube-api-access-m42mk\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.701728 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.701709    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.784814 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.784724    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:00.868233 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.868184    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" event={"ID":"01570eb7-f2dd-42fc-8b61-35555c4c855e","Type":"ContainerStarted","Data":"09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117"}
Nov 03 18:51:00.868396 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.868260    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" event={"ID":"01570eb7-f2dd-42fc-8b61-35555c4c855e","Type":"ContainerStarted","Data":"22c8aaf87d016545becdfce965ac29f50d7086acfb2da165336a88ccb0ebb5e0"}
Nov 03 18:51:00.869821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.869798    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" event={"ID":"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22","Type":"ContainerStarted","Data":"92b017934b83edd560a802dbf756cdf0c0605650059e0594ccfd0ebf5b48e349"}
Nov 03 18:51:00.869917 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.869836    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" event={"ID":"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22","Type":"ContainerStarted","Data":"7bcdfa4c82734c85daf26bdaa9572659a924dfc86c76ccf957dd7f48ab285a40"}
Nov 03 18:51:00.995853 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:00.995803    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podStartSLOduration=1.995789695 podStartE2EDuration="1.995789695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:00.995304656 +0000 UTC m=+790.143999249" watchObservedRunningTime="2026-11-03 18:51:00.995789695 +0000 UTC m=+790.144484265"
Nov 03 18:51:01.163914 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:01.163881    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e4111b_5349_4747_b39d_adba3dd53310.slice/crio-d69399cc228685d0afe70d0e94f71ede04ee65bf4e4af0ba1eaa5b816ff92761 WatchSource:0}: Error finding container d69399cc228685d0afe70d0e94f71ede04ee65bf4e4af0ba1eaa5b816ff92761: Status 404 returned error can't find the container with id d69399cc228685d0afe70d0e94f71ede04ee65bf4e4af0ba1eaa5b816ff92761
Nov 03 18:51:01.166422 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.166387    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:01.166505 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.166457    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:01.166505 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.166499    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:01.167168 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.167125    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"]
Nov 03 18:51:01.179421 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.179377    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" podStartSLOduration=2.179362152 podStartE2EDuration="2.179362152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:01.178306544 +0000 UTC m=+790.327001115" watchObservedRunningTime="2026-11-03 18:51:01.179362152 +0000 UTC m=+790.328056720"
Nov 03 18:51:01.545766 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.545680    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:01.552064 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.552036    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:01.849885 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.849797    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:01.854599 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.854573    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:01.875833 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.875800    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" event={"ID":"47e4111b-5349-4747-b39d-adba3dd53310","Type":"ContainerStarted","Data":"61d6095d50e5835857cdf9516972885823ef2a61d364cf78926e5e7994866c15"}
Nov 03 18:51:01.876010 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.875857    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" event={"ID":"47e4111b-5349-4747-b39d-adba3dd53310","Type":"ContainerStarted","Data":"d69399cc228685d0afe70d0e94f71ede04ee65bf4e4af0ba1eaa5b816ff92761"}
Nov 03 18:51:01.876133 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.876113    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:01.876363 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.876341    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:01.877007 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.876986    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:01.877117 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:01.877102    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:02.107086 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:02.106982    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" podStartSLOduration=2.106968322 podStartE2EDuration="2.106968322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:02.106346655 +0000 UTC m=+791.255041228" watchObservedRunningTime="2026-11-03 18:51:02.106968322 +0000 UTC m=+791.255662893"
Nov 03 18:51:02.785806 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:02.785766    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:02.790838 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:02.790812    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:02.880731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:02.880701    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:02.881630 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:02.881604    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:51:14.468960 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.468921    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"]
Nov 03 18:51:14.474389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.474366    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.527550 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.527513    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"]
Nov 03 18:51:14.574721 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.574689    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.574875 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.574813    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.574875 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.574843    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4n8n\" (UniqueName: \"kubernetes.io/projected/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-kube-api-access-n4n8n\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.574875 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.574864    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-metrics-certs\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.576861 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.576841    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-a4cacd-serving-cert\""
Nov 03 18:51:14.675489 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.675457    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.675660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.675567    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.675660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.675594    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n8n\" (UniqueName: \"kubernetes.io/projected/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-kube-api-access-n4n8n\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.675660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.675621    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-metrics-certs\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.675844 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:14.675676    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-serving-cert: secret "maas-api-e2e-worker-w3-a4cacd-serving-cert" not found
Nov 03 18:51:14.675844 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:14.675777    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls podName:101232cf-aba1-4fef-87ee-4b66ff8f2eeb nodeName:}" failed. No retries permitted until 2026-11-03 18:51:15.175755755 +0000 UTC m=+804.324450320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls") pod "maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9" (UID: "101232cf-aba1-4fef-87ee-4b66ff8f2eeb") : secret "maas-api-e2e-worker-w3-a4cacd-serving-cert" not found
Nov 03 18:51:14.676203 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.676174    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.678154 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.678124    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-metrics-certs\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:14.752082 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:14.752004    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4n8n\" (UniqueName: \"kubernetes.io/projected/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-kube-api-access-n4n8n\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:15.180876 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:15.180844    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:15.183505 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:15.183473    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:15.388701 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:15.388667    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:15.769534 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:15.769505    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101232cf_aba1_4fef_87ee_4b66ff8f2eeb.slice/crio-b2e5d245c5f556905eb4d91a2d3b90f9de629814723156b67a99bfd71be7fa34 WatchSource:0}: Error finding container b2e5d245c5f556905eb4d91a2d3b90f9de629814723156b67a99bfd71be7fa34: Status 404 returned error can't find the container with id b2e5d245c5f556905eb4d91a2d3b90f9de629814723156b67a99bfd71be7fa34
Nov 03 18:51:15.772671 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:15.772651    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"]
Nov 03 18:51:15.936612 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:15.936572    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9" event={"ID":"101232cf-aba1-4fef-87ee-4b66ff8f2eeb","Type":"ContainerStarted","Data":"b2e5d245c5f556905eb4d91a2d3b90f9de629814723156b67a99bfd71be7fa34"}
Nov 03 18:51:16.942609 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:16.942578    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9" event={"ID":"101232cf-aba1-4fef-87ee-4b66ff8f2eeb","Type":"ContainerStarted","Data":"99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21"}
Nov 03 18:51:16.943030 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:16.942669    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:17.066107 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:17.066054    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9" podStartSLOduration=2.238683514 podStartE2EDuration="3.066040139s" totalImagesPullingTime="827.356625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:17.064575294 +0000 UTC m=+806.213269882" watchObservedRunningTime="2026-11-03 18:51:17.066040139 +0000 UTC m=+806.214734710"
Nov 03 18:51:17.531975 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:17.531928    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:51:19.641415 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.641379    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"]
Nov 03 18:51:19.645012 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.644994    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.699323 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.699299    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-355e1a-serving-cert\""
Nov 03 18:51:19.701351 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.701291    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"]
Nov 03 18:51:19.727848 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.727818    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6305a-25c1-4534-aaec-0dc76184c358-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.727938 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.727860    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-metrics-certs\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.727978 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.727964    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.728110 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.728090    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhrz\" (UniqueName: \"kubernetes.io/projected/d4d6305a-25c1-4534-aaec-0dc76184c358-kube-api-access-zjhrz\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.829680 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.829639    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhrz\" (UniqueName: \"kubernetes.io/projected/d4d6305a-25c1-4534-aaec-0dc76184c358-kube-api-access-zjhrz\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.829869 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.829702    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6305a-25c1-4534-aaec-0dc76184c358-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.829869 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.829743    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-metrics-certs\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.829869 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.829811    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.830024 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:19.829945    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-serving-cert: secret "maas-api-e2e-worker-w1-355e1a-serving-cert" not found
Nov 03 18:51:19.830081 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:19.830025    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls podName:d4d6305a-25c1-4534-aaec-0dc76184c358 nodeName:}" failed. No retries permitted until 2026-11-03 18:51:20.33000655 +0000 UTC m=+809.478701113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls") pod "maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr" (UID: "d4d6305a-25c1-4534-aaec-0dc76184c358") : secret "maas-api-e2e-worker-w1-355e1a-serving-cert" not found
Nov 03 18:51:19.830567 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.830530    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6305a-25c1-4534-aaec-0dc76184c358-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.832755 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.832725    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-metrics-certs\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:19.895777 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:19.895713    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhrz\" (UniqueName: \"kubernetes.io/projected/d4d6305a-25c1-4534-aaec-0dc76184c358-kube-api-access-zjhrz\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:20.335104 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:20.335065    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:20.337627 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:20.337599    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls\") pod \"maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:20.557304 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:20.557271    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:20.927643 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:20.927612    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d6305a_25c1_4534_aaec_0dc76184c358.slice/crio-f3a220817521fbf6529e80d2f5b058612920412b00deb65f0fafabc3b302b65c WatchSource:0}: Error finding container f3a220817521fbf6529e80d2f5b058612920412b00deb65f0fafabc3b302b65c: Status 404 returned error can't find the container with id f3a220817521fbf6529e80d2f5b058612920412b00deb65f0fafabc3b302b65c
Nov 03 18:51:20.931508 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:20.931482    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"]
Nov 03 18:51:20.963852 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:20.963821    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr" event={"ID":"d4d6305a-25c1-4534-aaec-0dc76184c358","Type":"ContainerStarted","Data":"f3a220817521fbf6529e80d2f5b058612920412b00deb65f0fafabc3b302b65c"}
Nov 03 18:51:21.970437 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:21.970400    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr" event={"ID":"d4d6305a-25c1-4534-aaec-0dc76184c358","Type":"ContainerStarted","Data":"ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc"}
Nov 03 18:51:21.970867 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:21.970548    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:22.115747 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:22.115646    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr" podStartSLOduration=2.226511298 podStartE2EDuration="3.115632606s" totalImagesPullingTime="889.121308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:22.114339126 +0000 UTC m=+811.263033698" watchObservedRunningTime="2026-11-03 18:51:22.115632606 +0000 UTC m=+811.264327176"
Nov 03 18:51:22.953381 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:22.953355    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 18:51:27.870169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:27.867475    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"]
Nov 03 18:51:27.875723 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:27.875696    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:27.912291 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:27.912261    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"]
Nov 03 18:51:27.966419 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:27.966388    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9686c3\""
Nov 03 18:51:27.966754 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:27.966735    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9686c3-dockercfg-4z4kc\""
Nov 03 18:51:27.979748 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:27.979718    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 18:51:28.011722 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.011694    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:28.011919 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.011824    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvs5k\" (UniqueName: \"kubernetes.io/projected/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-kube-api-access-fvs5k\") pod \"payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:28.112514 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.112476    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvs5k\" (UniqueName: \"kubernetes.io/projected/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-kube-api-access-fvs5k\") pod \"payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:28.112718 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.112528    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:28.113171 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.113132    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:28.182822 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.182789    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvs5k\" (UniqueName: \"kubernetes.io/projected/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-kube-api-access-fvs5k\") pod \"payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:28.188230 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.188206    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:28.568339 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:28.566402    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"]
Nov 03 18:51:28.575886 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:28.575855    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2e4eaf_8cea_46bb_9327_9bce2f88e081.slice/crio-779ca4bd7108b69e0969516ca5df3be0e3338c7d5d01ad300b70f5140ccb64e8 WatchSource:0}: Error finding container 779ca4bd7108b69e0969516ca5df3be0e3338c7d5d01ad300b70f5140ccb64e8: Status 404 returned error can't find the container with id 779ca4bd7108b69e0969516ca5df3be0e3338c7d5d01ad300b70f5140ccb64e8
Nov 03 18:51:29.003793 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:29.003758    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" event={"ID":"7a2e4eaf-8cea-46bb-9327-9bce2f88e081","Type":"ContainerStarted","Data":"779ca4bd7108b69e0969516ca5df3be0e3338c7d5d01ad300b70f5140ccb64e8"}
Nov 03 18:51:32.029279 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.029238    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" event={"ID":"7a2e4eaf-8cea-46bb-9327-9bce2f88e081","Type":"ContainerStarted","Data":"0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab"}
Nov 03 18:51:32.029775 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.029375    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:32.164454 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.164388    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" podStartSLOduration=2.505968768 podStartE2EDuration="5.164374216s" totalImagesPullingTime="2.658405448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:32.149520802 +0000 UTC m=+821.298215373" watchObservedRunningTime="2026-11-03 18:51:32.164374216 +0000 UTC m=+821.313068787"
Nov 03 18:51:32.877356 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.877316    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"]
Nov 03 18:51:32.883396 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.883366    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:32.890329 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.890300    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec70e8-d473-4799-ac6d-79fa07e44c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:32.890443 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.890358    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ctd\" (UniqueName: \"kubernetes.io/projected/1bec70e8-d473-4799-ac6d-79fa07e44c99-kube-api-access-f9ctd\") pod \"payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:32.918317 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.918271    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:51:32.918497 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.918378    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:51:32.938125 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.938100    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-3a7056\""
Nov 03 18:51:32.938302 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.938127    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-3a7056-dockercfg-w9bqx\""
Nov 03 18:51:32.982311 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.982273    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"]
Nov 03 18:51:32.991336 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.991302    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec70e8-d473-4799-ac6d-79fa07e44c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:32.991571 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.991354    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ctd\" (UniqueName: \"kubernetes.io/projected/1bec70e8-d473-4799-ac6d-79fa07e44c99-kube-api-access-f9ctd\") pod \"payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:32.992231 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:32.992207    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec70e8-d473-4799-ac6d-79fa07e44c99-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:33.085856 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:33.085229    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ctd\" (UniqueName: \"kubernetes.io/projected/1bec70e8-d473-4799-ac6d-79fa07e44c99-kube-api-access-f9ctd\") pod \"payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:33.196362 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:33.196321    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:33.579874 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:33.579844    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bec70e8_d473_4799_ac6d_79fa07e44c99.slice/crio-9ba5dc969985b9d4239026b7d1fe9cc9e06b2d9c95dc507bbfb43771c33ef4c8 WatchSource:0}: Error finding container 9ba5dc969985b9d4239026b7d1fe9cc9e06b2d9c95dc507bbfb43771c33ef4c8: Status 404 returned error can't find the container with id 9ba5dc969985b9d4239026b7d1fe9cc9e06b2d9c95dc507bbfb43771c33ef4c8
Nov 03 18:51:33.580800 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:33.580740    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"]
Nov 03 18:51:34.041178 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.041078    4489 generic.go:356] "Generic (PLEG): container finished" podID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerID="09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117" exitCode=0
Nov 03 18:51:34.041388 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.041173    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" event={"ID":"01570eb7-f2dd-42fc-8b61-35555c4c855e","Type":"ContainerDied","Data":"09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117"}
Nov 03 18:51:34.041643 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.041626    4489 scope.go:122] "RemoveContainer" containerID="09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117"
Nov 03 18:51:34.042765 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.042743    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" event={"ID":"1bec70e8-d473-4799-ac6d-79fa07e44c99","Type":"ContainerStarted","Data":"a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b"}
Nov 03 18:51:34.042868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.042786    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" event={"ID":"1bec70e8-d473-4799-ac6d-79fa07e44c99","Type":"ContainerStarted","Data":"9ba5dc969985b9d4239026b7d1fe9cc9e06b2d9c95dc507bbfb43771c33ef4c8"}
Nov 03 18:51:34.042868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.042819    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:34.042984 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.042949    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:34.043033 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.043010    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:34.043083 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.043042    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:34.171169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.170489    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"]
Nov 03 18:51:34.181702 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.180440    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.195949 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.195919    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"]
Nov 03 18:51:34.205644 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.205616    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7755b4-3a94-46eb-8ff4-3c651c11c230-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.205778 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.205659    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.205915 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.205889    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-metrics-certs\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.205953 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.205936    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrqp\" (UniqueName: \"kubernetes.io/projected/fe7755b4-3a94-46eb-8ff4-3c651c11c230-kube-api-access-bzrqp\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.237022 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.236988    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-9686c3-serving-cert\""
Nov 03 18:51:34.307897 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.307842    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-metrics-certs\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.308056 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.307916    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrqp\" (UniqueName: \"kubernetes.io/projected/fe7755b4-3a94-46eb-8ff4-3c651c11c230-kube-api-access-bzrqp\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.308056 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.308020    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7755b4-3a94-46eb-8ff4-3c651c11c230-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.308056 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.308044    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.308311 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:34.308293    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-serving-cert: secret "maas-api-e2e-worker-w0-9686c3-serving-cert" not found
Nov 03 18:51:34.308394 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:34.308381    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls podName:fe7755b4-3a94-46eb-8ff4-3c651c11c230 nodeName:}" failed. No retries permitted until 2026-11-03 18:51:34.808360159 +0000 UTC m=+823.957054730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls") pod "maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt" (UID: "fe7755b4-3a94-46eb-8ff4-3c651c11c230") : secret "maas-api-e2e-worker-w0-9686c3-serving-cert" not found
Nov 03 18:51:34.309118 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.309096    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7755b4-3a94-46eb-8ff4-3c651c11c230-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.312884 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.312860    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-metrics-certs\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.379902 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.379799    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" podStartSLOduration=2.379781301 podStartE2EDuration="2.379781301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:34.363847786 +0000 UTC m=+823.512542359" watchObservedRunningTime="2026-11-03 18:51:34.379781301 +0000 UTC m=+823.528475874"
Nov 03 18:51:34.382382 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.382357    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrqp\" (UniqueName: \"kubernetes.io/projected/fe7755b4-3a94-46eb-8ff4-3c651c11c230-kube-api-access-bzrqp\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.813041 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.813005    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:34.815640 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:34.815620    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:35.062166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:35.058635    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" event={"ID":"01570eb7-f2dd-42fc-8b61-35555c4c855e","Type":"ContainerStarted","Data":"ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405"}
Nov 03 18:51:35.099535 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:35.099432    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:35.506167 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:35.505474    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"]
Nov 03 18:51:35.512786 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:35.512470    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7755b4_3a94_46eb_8ff4_3c651c11c230.slice/crio-331c76a24d3bb6271e3b2b90c20bf32edf54e26b730b67af588cd886338f3e1e WatchSource:0}: Error finding container 331c76a24d3bb6271e3b2b90c20bf32edf54e26b730b67af588cd886338f3e1e: Status 404 returned error can't find the container with id 331c76a24d3bb6271e3b2b90c20bf32edf54e26b730b67af588cd886338f3e1e
Nov 03 18:51:35.546011 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:35.545984    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:35.546235 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:35.546216    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:51:35.546292 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:35.546259    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 18:51:36.092766 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:36.092694    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt" event={"ID":"fe7755b4-3a94-46eb-8ff4-3c651c11c230","Type":"ContainerStarted","Data":"331c76a24d3bb6271e3b2b90c20bf32edf54e26b730b67af588cd886338f3e1e"}
Nov 03 18:51:36.545535 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:36.545500    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:51:36.545888 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:36.545552    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 18:51:37.111194 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:37.111128    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt" event={"ID":"fe7755b4-3a94-46eb-8ff4-3c651c11c230","Type":"ContainerStarted","Data":"2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e"}
Nov 03 18:51:37.112017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:37.112000    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:37.236941 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:37.236884    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt" podStartSLOduration=2.237140464 podStartE2EDuration="3.23687069s" totalImagesPullingTime="999.730226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:37.234890222 +0000 UTC m=+826.383584792" watchObservedRunningTime="2026-11-03 18:51:37.23687069 +0000 UTC m=+826.385565260"
Nov 03 18:51:37.546134 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:37.546099    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:51:37.546554 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:37.546197    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 18:51:38.546039 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:38.546005    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:51:38.546223 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:38.546063    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 18:51:39.546164 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:39.546102    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:51:39.546348 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:39.546203    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 18:51:40.549704 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:40.549676    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:40.550079 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:40.549989    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:40.550722 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:40.550701    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 18:51:42.136300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:42.136205    4489 generic.go:356] "Generic (PLEG): container finished" podID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerID="92b017934b83edd560a802dbf756cdf0c0605650059e0594ccfd0ebf5b48e349" exitCode=0
Nov 03 18:51:42.136300 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:42.136278    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" event={"ID":"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22","Type":"ContainerDied","Data":"92b017934b83edd560a802dbf756cdf0c0605650059e0594ccfd0ebf5b48e349"}
Nov 03 18:51:42.136757 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:42.136721    4489 scope.go:122] "RemoveContainer" containerID="92b017934b83edd560a802dbf756cdf0c0605650059e0594ccfd0ebf5b48e349"
Nov 03 18:51:42.137921 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:42.137894    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:42.138008 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:42.137954    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:42.138008 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:42.137989    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:51:43.037996 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:43.037969    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 18:51:43.142673 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:43.142636    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" event={"ID":"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22","Type":"ContainerStarted","Data":"3a91f2f197df392543f6bad4de086fc85e104f4bd187cb02d061b8ae02ea7cf6"}
Nov 03 18:51:43.853382 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:43.853346    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:43.853741 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:43.853719    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 18:51:43.853922 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:43.853893    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 18:51:44.126977 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:44.126904    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 18:51:44.868169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:44.861535    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 18:51:44.868169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:44.861598    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 18:51:45.074189 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:45.074125    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 18:51:45.872169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:45.871218    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 18:51:45.872169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:45.871310    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 18:51:46.859169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:46.858486    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 18:51:46.859169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:46.858545    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 18:51:46.876461 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:46.876435    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:47.529956 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:51:47.529912    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:51:47.870227 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:47.869587    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 18:51:47.870227 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:47.869688    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 18:51:48.898229 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:48.898197    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:48.900048 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:48.900023    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 18:51:53.485547 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.485514    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"]
Nov 03 18:51:53.488676 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.488650    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.546657 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.546623    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"]
Nov 03 18:51:53.595957 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.595929    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-3a7056-serving-cert\""
Nov 03 18:51:53.647848 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.647813    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f904d-4a17-4e30-95e9-8e343a5f613f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.648026 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.647882    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-metrics-certs\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.648026 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.647939    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67w5v\" (UniqueName: \"kubernetes.io/projected/126f904d-4a17-4e30-95e9-8e343a5f613f-kube-api-access-67w5v\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.648166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.648035    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.749568 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.749477    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f904d-4a17-4e30-95e9-8e343a5f613f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.749568 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.749543    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-metrics-certs\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.749568 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.749566    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67w5v\" (UniqueName: \"kubernetes.io/projected/126f904d-4a17-4e30-95e9-8e343a5f613f-kube-api-access-67w5v\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.749834 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.749657    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.750263 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.750238    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f904d-4a17-4e30-95e9-8e343a5f613f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.752402 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.752379    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-metrics-certs\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.752511 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.752449    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:53.813208 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:53.813136    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67w5v\" (UniqueName: \"kubernetes.io/projected/126f904d-4a17-4e30-95e9-8e343a5f613f-kube-api-access-67w5v\") pod \"maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:54.101069 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:54.100979    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:54.478346 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:51:54.478303    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126f904d_4a17_4e30_95e9_8e343a5f613f.slice/crio-7723dcac2010a2e405b8c0108cdb49b1dec5dc68c139cf11e19c6de7adc1a6ef WatchSource:0}: Error finding container 7723dcac2010a2e405b8c0108cdb49b1dec5dc68c139cf11e19c6de7adc1a6ef: Status 404 returned error can't find the container with id 7723dcac2010a2e405b8c0108cdb49b1dec5dc68c139cf11e19c6de7adc1a6ef
Nov 03 18:51:54.480192 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:54.480169    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"]
Nov 03 18:51:55.221368 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:55.221333    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66" event={"ID":"126f904d-4a17-4e30-95e9-8e343a5f613f","Type":"ContainerStarted","Data":"7723dcac2010a2e405b8c0108cdb49b1dec5dc68c139cf11e19c6de7adc1a6ef"}
Nov 03 18:51:56.226971 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:56.226937    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66" event={"ID":"126f904d-4a17-4e30-95e9-8e343a5f613f","Type":"ContainerStarted","Data":"4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a"}
Nov 03 18:51:56.227386 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:56.227046    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:51:56.349802 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:51:56.349754    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66" podStartSLOduration=2.482506608 podStartE2EDuration="3.349741759s" totalImagesPullingTime="867.235151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:56.349500449 +0000 UTC m=+845.498195024" watchObservedRunningTime="2026-11-03 18:51:56.349741759 +0000 UTC m=+845.498436329"
Nov 03 18:52:01.975581 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:01.975489    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"]
Nov 03 18:52:01.978136 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:01.978114    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.027181 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.027123    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-home\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.027348 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.027264    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bnr\" (UniqueName: \"kubernetes.io/projected/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-kube-api-access-97bnr\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.027348 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.027327    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tls-certs\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.027439 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.027349    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-model-cache\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.027439 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.027370    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tmp-dir\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.027516 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.027450    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-dshm\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.034218 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.034187    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"]
Nov 03 18:52:02.083693 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.083658    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"simulator-w3-a4cacd-kserve-self-signed-certs\""
Nov 03 18:52:02.084000 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.083980    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"default-dockercfg-vzdn4\""
Nov 03 18:52:02.084807 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.084782    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"kube-root-ca.crt\""
Nov 03 18:52:02.084807 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.084794    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"openshift-service-ca.crt\""
Nov 03 18:52:02.128187 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128127    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bnr\" (UniqueName: \"kubernetes.io/projected/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-kube-api-access-97bnr\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128225    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tls-certs\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128246    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-model-cache\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128262    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tmp-dir\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128284    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-dshm\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128372 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128329    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-home\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128720 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128699    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tmp-dir\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128791 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128697    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-model-cache\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.128791 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.128750    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-home\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.130801 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.130777    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-dshm\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.131035 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.131016    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tls-certs\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.191669 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.191630    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bnr\" (UniqueName: \"kubernetes.io/projected/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-kube-api-access-97bnr\") pod \"simulator-w3-a4cacd-kserve-67885cb984-w2lfx\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") " pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.236962 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.236882    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 18:52:02.288831 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.288791    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:02.658025 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:52:02.657996    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba917b5c_a85a_4afd_a29c_4d4b6a5ce1a8.slice/crio-5a1fe2b1496e59ec779ebbf638849f292f1e8005c5100ca5aeb7f273044fc4e1 WatchSource:0}: Error finding container 5a1fe2b1496e59ec779ebbf638849f292f1e8005c5100ca5aeb7f273044fc4e1: Status 404 returned error can't find the container with id 5a1fe2b1496e59ec779ebbf638849f292f1e8005c5100ca5aeb7f273044fc4e1
Nov 03 18:52:02.661096 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:02.661075    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"]
Nov 03 18:52:03.272917 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:03.272880    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" event={"ID":"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8","Type":"ContainerStarted","Data":"cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a"}
Nov 03 18:52:03.273568 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:03.273543    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" event={"ID":"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8","Type":"ContainerStarted","Data":"5a1fe2b1496e59ec779ebbf638849f292f1e8005c5100ca5aeb7f273044fc4e1"}
Nov 03 18:52:03.289432 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:03.289226    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:03.398604 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:03.398528    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podStartSLOduration=2.398508124 podStartE2EDuration="2.398508124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:03.396379318 +0000 UTC m=+852.545073890" watchObservedRunningTime="2026-11-03 18:52:03.398508124 +0000 UTC m=+852.547202696"
Nov 03 18:52:12.289422 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:12.289390    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:12.304001 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:12.302913    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:12.335314 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:12.335278    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 18:52:13.325286 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:13.325255    4489 generic.go:356] "Generic (PLEG): container finished" podID="47e4111b-5349-4747-b39d-adba3dd53310" containerID="61d6095d50e5835857cdf9516972885823ef2a61d364cf78926e5e7994866c15" exitCode=0
Nov 03 18:52:13.325702 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:13.325332    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" event={"ID":"47e4111b-5349-4747-b39d-adba3dd53310","Type":"ContainerDied","Data":"61d6095d50e5835857cdf9516972885823ef2a61d364cf78926e5e7994866c15"}
Nov 03 18:52:13.325772 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:13.325759    4489 scope.go:122] "RemoveContainer" containerID="61d6095d50e5835857cdf9516972885823ef2a61d364cf78926e5e7994866c15"
Nov 03 18:52:13.326983 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:13.326961    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:52:13.327041 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:13.327024    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:52:13.327075 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:13.327054    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:52:14.333686 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:14.333647    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" event={"ID":"47e4111b-5349-4747-b39d-adba3dd53310","Type":"ContainerStarted","Data":"24b8769c99134aade74c9c926cf789eda4d1d095b034538c1c1822e1be759129"}
Nov 03 18:52:14.785582 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:14.785539    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:52:14.787047 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:14.787015    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 18:52:14.787205 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:14.787075    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 18:52:15.785472 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:15.785432    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 18:52:15.786000 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:15.785496    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 18:52:16.785757 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:16.785719    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 18:52:16.786194 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:16.785775    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 18:52:17.529374 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:52:17.529338    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:52:17.785923 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:17.785831    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 18:52:17.785923 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:17.785891    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 18:52:17.881505 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:17.881468    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:52:18.785451 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:18.785417    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 18:52:18.785626 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:18.785475    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 18:52:19.790565 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:19.790528    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:52:19.791485 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:52:19.791465    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 18:52:47.530245 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:52:47.530206    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:53:17.529960 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:53:17.529926    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:53:25.911199 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:25.911169    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"]
Nov 03 18:53:25.915017 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:25.915003    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:25.971384 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:25.971352    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"]
Nov 03 18:53:26.020208 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020178    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-tmp-dir\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.020371 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020286    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-home\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.020371 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020314    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8j6w\" (UniqueName: \"kubernetes.io/projected/565d86a6-1df3-4a4d-aefd-44369e76b11e-kube-api-access-f8j6w\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.020446 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020409    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-model-cache\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.020665 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020645    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-dshm\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.020789 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020768    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565d86a6-1df3-4a4d-aefd-44369e76b11e-tls-certs\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.021013 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020996    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"openshift-service-ca.crt\""
Nov 03 18:53:26.021013 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.021009    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"default-dockercfg-zr88h\""
Nov 03 18:53:26.021135 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.020996    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"kube-root-ca.crt\""
Nov 03 18:53:26.022497 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.022482    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"simulator-w1-355e1a-kserve-self-signed-certs\""
Nov 03 18:53:26.122061 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122027    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-dshm\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122061 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122067    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565d86a6-1df3-4a4d-aefd-44369e76b11e-tls-certs\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122337 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122122    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-tmp-dir\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122337 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122217    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-home\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122337 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122241    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8j6w\" (UniqueName: \"kubernetes.io/projected/565d86a6-1df3-4a4d-aefd-44369e76b11e-kube-api-access-f8j6w\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122337 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122291    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-model-cache\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122596 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122567    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-home\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122735 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122638    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-model-cache\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.122735 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.122696    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-tmp-dir\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.124485 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.124468    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-dshm\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.124760 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.124743    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565d86a6-1df3-4a4d-aefd-44369e76b11e-tls-certs\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.185868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.185797    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8j6w\" (UniqueName: \"kubernetes.io/projected/565d86a6-1df3-4a4d-aefd-44369e76b11e-kube-api-access-f8j6w\") pod \"simulator-w1-355e1a-kserve-77dd4476-s46jv\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") " pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.226748 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.226713    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:26.593973 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:53:26.593944    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod565d86a6_1df3_4a4d_aefd_44369e76b11e.slice/crio-ed9a3ab18d2a8a7d119f02e06a8b4674170412e9e98fb357844f730050d637ff WatchSource:0}: Error finding container ed9a3ab18d2a8a7d119f02e06a8b4674170412e9e98fb357844f730050d637ff: Status 404 returned error can't find the container with id ed9a3ab18d2a8a7d119f02e06a8b4674170412e9e98fb357844f730050d637ff
Nov 03 18:53:26.597443 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.597419    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"]
Nov 03 18:53:26.699187 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.699118    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" event={"ID":"565d86a6-1df3-4a4d-aefd-44369e76b11e","Type":"ContainerStarted","Data":"38a36ac2cb4e7b108bb7bf7a05e74b49b63b79c551e98985d1089701f1be6b10"}
Nov 03 18:53:26.699187 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.699190    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" event={"ID":"565d86a6-1df3-4a4d-aefd-44369e76b11e","Type":"ContainerStarted","Data":"ed9a3ab18d2a8a7d119f02e06a8b4674170412e9e98fb357844f730050d637ff"}
Nov 03 18:53:26.823442 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:26.823388    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podStartSLOduration=1.823374699 podStartE2EDuration="1.823374699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:26.821650749 +0000 UTC m=+935.970345321" watchObservedRunningTime="2026-11-03 18:53:26.823374699 +0000 UTC m=+935.972069270"
Nov 03 18:53:27.227805 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:27.227758    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:36.227335 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:36.227249    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:36.239921 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:36.239895    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:36.766273 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:36.766244    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 18:53:44.963518 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:44.963481    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"]
Nov 03 18:53:44.967264 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:44.967247    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.025805 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.025768    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"]
Nov 03 18:53:45.074359 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.074321    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"default-dockercfg-6l242\""
Nov 03 18:53:45.074553 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.074435    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"simulator-w0-9686c3-kserve-self-signed-certs\""
Nov 03 18:53:45.076063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.076042    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"kube-root-ca.crt\""
Nov 03 18:53:45.076063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.076051    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"openshift-service-ca.crt\""
Nov 03 18:53:45.112407 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.112366    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tmp-dir\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.112732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.112494    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-home\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.112732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.112555    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tls-certs\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.112732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.112585    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmqgv\" (UniqueName: \"kubernetes.io/projected/74ea7133-1b1f-44ab-a003-ff63a40a72f9-kube-api-access-jmqgv\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.112732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.112614    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-model-cache\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.112732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.112653    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-dshm\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213424 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213386    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tmp-dir\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213483    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-home\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213523    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tls-certs\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213542    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmqgv\" (UniqueName: \"kubernetes.io/projected/74ea7133-1b1f-44ab-a003-ff63a40a72f9-kube-api-access-jmqgv\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213561    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-model-cache\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213583    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-dshm\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213931 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213834    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tmp-dir\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.213990 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213930    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-home\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.214042 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.213989    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-model-cache\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.216189 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.216162    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-dshm\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.216303 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.216256    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tls-certs\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.280367 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.280338    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmqgv\" (UniqueName: \"kubernetes.io/projected/74ea7133-1b1f-44ab-a003-ff63a40a72f9-kube-api-access-jmqgv\") pod \"simulator-w0-9686c3-kserve-57cbb49666-8tx49\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") " pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.580131 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.580037    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:45.957321 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:53:45.957289    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74ea7133_1b1f_44ab_a003_ff63a40a72f9.slice/crio-f151a1e465002f4a1bee0670e6efe3babb8c9fd02f719c81cbec8143d5b6dd43 WatchSource:0}: Error finding container f151a1e465002f4a1bee0670e6efe3babb8c9fd02f719c81cbec8143d5b6dd43: Status 404 returned error can't find the container with id f151a1e465002f4a1bee0670e6efe3babb8c9fd02f719c81cbec8143d5b6dd43
Nov 03 18:53:45.959856 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:45.959764    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"]
Nov 03 18:53:46.806853 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:46.806818    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" event={"ID":"74ea7133-1b1f-44ab-a003-ff63a40a72f9","Type":"ContainerStarted","Data":"995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26"}
Nov 03 18:53:46.806853 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:46.806859    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" event={"ID":"74ea7133-1b1f-44ab-a003-ff63a40a72f9","Type":"ContainerStarted","Data":"f151a1e465002f4a1bee0670e6efe3babb8c9fd02f719c81cbec8143d5b6dd43"}
Nov 03 18:53:46.930126 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:46.930073    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podStartSLOduration=2.930058871 podStartE2EDuration="2.930058871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:46.92921208 +0000 UTC m=+956.077906651" watchObservedRunningTime="2026-11-03 18:53:46.930058871 +0000 UTC m=+956.078753442"
Nov 03 18:53:47.531189 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:53:47.531126    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:53:47.580560 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:47.580526    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:55.580771 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:55.580732    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:55.593161 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:55.593125    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:53:55.860839 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:53:55.860759    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 18:54:17.530643 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:54:17.530608    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:54:33.963360 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:33.963322    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"]
Nov 03 18:54:33.967052 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:33.967035    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.022356 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.022326    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"]
Nov 03 18:54:34.071996 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.071966    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"kube-root-ca.crt\""
Nov 03 18:54:34.074342 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.074315    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"premium-w5-3a7056-kserve-self-signed-certs\""
Nov 03 18:54:34.074342 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.074316    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"openshift-service-ca.crt\""
Nov 03 18:54:34.074342 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.074325    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"default-dockercfg-h7sxm\""
Nov 03 18:54:34.119356 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.119321    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-tmp-dir\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.119557 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.119399    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635f4eda-d081-47bc-9f53-25559992ea53-tls-certs\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.119557 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.119532    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-model-cache\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.119709 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.119637    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-dshm\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.119709 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.119677    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbxx\" (UniqueName: \"kubernetes.io/projected/635f4eda-d081-47bc-9f53-25559992ea53-kube-api-access-klbxx\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.119709 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.119700    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-home\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.220586 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.220502    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-tmp-dir\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.220586 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.220559    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635f4eda-d081-47bc-9f53-25559992ea53-tls-certs\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.220773 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.220636    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-model-cache\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.220773 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.220702    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-dshm\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.220773 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.220753    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbxx\" (UniqueName: \"kubernetes.io/projected/635f4eda-d081-47bc-9f53-25559992ea53-kube-api-access-klbxx\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.220925 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.220788    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-home\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.220988 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.220962    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-tmp-dir\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.221070 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.221017    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-model-cache\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.221185 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.221165    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-home\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.222989 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.222966    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-dshm\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.223177 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.223128    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635f4eda-d081-47bc-9f53-25559992ea53-tls-certs\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.284012 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.283962    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbxx\" (UniqueName: \"kubernetes.io/projected/635f4eda-d081-47bc-9f53-25559992ea53-kube-api-access-klbxx\") pod \"premium-w5-3a7056-kserve-7db767c5c7-b797s\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") " pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.578762 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.578682    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:34.947921 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:54:34.947895    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod635f4eda_d081_47bc_9f53_25559992ea53.slice/crio-7e207f42b2b042aa01d1f5429e98d3f40d8d04d6bea488a0af9a58d675fc6054 WatchSource:0}: Error finding container 7e207f42b2b042aa01d1f5429e98d3f40d8d04d6bea488a0af9a58d675fc6054: Status 404 returned error can't find the container with id 7e207f42b2b042aa01d1f5429e98d3f40d8d04d6bea488a0af9a58d675fc6054
Nov 03 18:54:34.951574 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:34.951549    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"]
Nov 03 18:54:35.038018 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:35.037986    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" event={"ID":"635f4eda-d081-47bc-9f53-25559992ea53","Type":"ContainerStarted","Data":"06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a"}
Nov 03 18:54:35.038420 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:35.038026    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" event={"ID":"635f4eda-d081-47bc-9f53-25559992ea53","Type":"ContainerStarted","Data":"7e207f42b2b042aa01d1f5429e98d3f40d8d04d6bea488a0af9a58d675fc6054"}
Nov 03 18:54:35.160933 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:35.160875    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podStartSLOduration=2.160859983 podStartE2EDuration="2.160859983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:35.158533831 +0000 UTC m=+1004.307228402" watchObservedRunningTime="2026-11-03 18:54:35.160859983 +0000 UTC m=+1004.309554602"
Nov 03 18:54:35.579475 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:35.579440    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:44.579384 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:44.579301    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:44.591666 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:44.591635    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:45.097687 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:54:45.097654    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 18:54:47.529532 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:54:47.529494    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:55:04.905780 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:04.905684    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"]
Nov 03 18:55:04.909437 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:04.909415    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:04.965190 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:04.965117    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"kube-root-ca.crt\""
Nov 03 18:55:04.967706 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:04.967673    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"]
Nov 03 18:55:05.015958 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.015922    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"openshift-service-ca.crt\""
Nov 03 18:55:05.017232 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.017206    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"default-dockercfg-zbhk5\""
Nov 03 18:55:05.017232 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.017212    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"premium-w2-32a59b-kserve-self-signed-certs\""
Nov 03 18:55:05.046945 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.046908    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-tmp-dir\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.046945 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.046946    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-dshm\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.047232 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.046993    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-home\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.047232 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.047058    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9ffp\" (UniqueName: \"kubernetes.io/projected/4d0751b0-b70c-4163-b158-7db0c3c9432e-kube-api-access-s9ffp\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.047232 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.047198    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0751b0-b70c-4163-b158-7db0c3c9432e-tls-certs\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.047393 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.047248    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-model-cache\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149058 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149000    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-tmp-dir\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149058 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149053    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-dshm\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149461 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149091    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-home\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149461 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149168    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9ffp\" (UniqueName: \"kubernetes.io/projected/4d0751b0-b70c-4163-b158-7db0c3c9432e-kube-api-access-s9ffp\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149461 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149218    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0751b0-b70c-4163-b158-7db0c3c9432e-tls-certs\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149461 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149258    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-model-cache\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149685 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149484    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-tmp-dir\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.149746 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.149702    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-model-cache\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.150168 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.150116    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-home\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.151813 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.151774    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-dshm\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.152692 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.152669    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0751b0-b70c-4163-b158-7db0c3c9432e-tls-certs\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.214364 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.214325    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9ffp\" (UniqueName: \"kubernetes.io/projected/4d0751b0-b70c-4163-b158-7db0c3c9432e-kube-api-access-s9ffp\") pod \"premium-w2-32a59b-kserve-6f48b9c564-t8vps\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") " pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.221794 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.221755    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:05.636174 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:55:05.636110    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d0751b0_b70c_4163_b158_7db0c3c9432e.slice/crio-ebc4b21852c8a97972c4dc0c786097248f49272760286f0980f7f201eca17b1f WatchSource:0}: Error finding container ebc4b21852c8a97972c4dc0c786097248f49272760286f0980f7f201eca17b1f: Status 404 returned error can't find the container with id ebc4b21852c8a97972c4dc0c786097248f49272760286f0980f7f201eca17b1f
Nov 03 18:55:05.639068 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:05.639037    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"]
Nov 03 18:55:06.191707 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:06.191674    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" event={"ID":"4d0751b0-b70c-4163-b158-7db0c3c9432e","Type":"ContainerStarted","Data":"61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"}
Nov 03 18:55:06.191707 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:06.191716    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" event={"ID":"4d0751b0-b70c-4163-b158-7db0c3c9432e","Type":"ContainerStarted","Data":"ebc4b21852c8a97972c4dc0c786097248f49272760286f0980f7f201eca17b1f"}
Nov 03 18:55:06.222639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:06.222603    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:06.314494 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:06.314435    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podStartSLOduration=2.314419187 podStartE2EDuration="2.314419187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:06.312856562 +0000 UTC m=+1035.461551133" watchObservedRunningTime="2026-11-03 18:55:06.314419187 +0000 UTC m=+1035.463113792"
Nov 03 18:55:15.228322 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:15.228258    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:15.241373 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:15.241348    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:15.268521 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:55:15.268496    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 18:55:17.530853 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:55:17.530821    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:55:47.534502 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:55:47.534465    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:56:17.530627 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:56:17.530576    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:56:31.513991 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.513911    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"]
Nov 03 18:56:31.517571 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.517551    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.610636 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.610594    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.610791 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.610655    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787654a0-d74e-40dd-9641-db7df5d589cd-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.610856 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.610814    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.610856 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.610842    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.610968 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.610884    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.611278 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.611253    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.611353 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.611327    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.611395 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.611351    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68cnd\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-kube-api-access-68cnd\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.611451 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.611436    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787654a0-d74e-40dd-9641-db7df5d589cd-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.628183 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.628156    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2e052a-openshift-default-dockercfg-vjkcv\""
Nov 03 18:56:31.630450 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.630425    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"]
Nov 03 18:56:31.712308 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712281    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712485 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712315    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712485 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712365    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712485 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712380    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68cnd\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-kube-api-access-68cnd\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712485 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712424    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787654a0-d74e-40dd-9641-db7df5d589cd-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712485 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712465    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712767 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712511    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787654a0-d74e-40dd-9641-db7df5d589cd-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712767 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712579    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712767 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712601    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712922 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712771    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.712922 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.712813    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.713105 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.713082    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.713349 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.713327    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.713505 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.713488    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787654a0-d74e-40dd-9641-db7df5d589cd-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.714770 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.714750    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.715613 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.715592    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787654a0-d74e-40dd-9641-db7df5d589cd-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.777525 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.777467    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.777635 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.777550    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68cnd\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-kube-api-access-68cnd\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:31.830079 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:31.830052    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:32.264161 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:56:32.264104    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787654a0_d74e_40dd_9641_db7df5d589cd.slice/crio-6fe33a0c6dcd22618994ee24ab82b8daaae830f31f6efe1e3b285cf802db1ec4 WatchSource:0}: Error finding container 6fe33a0c6dcd22618994ee24ab82b8daaae830f31f6efe1e3b285cf802db1ec4: Status 404 returned error can't find the container with id 6fe33a0c6dcd22618994ee24ab82b8daaae830f31f6efe1e3b285cf802db1ec4
Nov 03 18:56:32.267073 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.267038    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:56:32.267204 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.267123    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:56:32.267283 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.267211    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:56:32.267467 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.267445    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"]
Nov 03 18:56:32.669580 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.669531    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" event={"ID":"787654a0-d74e-40dd-9641-db7df5d589cd","Type":"ContainerStarted","Data":"1ba32c925cbd191f35208cb5d4de3907bc9b8b87ce28d2539fdf769216b7b248"}
Nov 03 18:56:32.670078 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.669634    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" event={"ID":"787654a0-d74e-40dd-9641-db7df5d589cd","Type":"ContainerStarted","Data":"6fe33a0c6dcd22618994ee24ab82b8daaae830f31f6efe1e3b285cf802db1ec4"}
Nov 03 18:56:32.795626 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.795571    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" podStartSLOduration=1.795558045 podStartE2EDuration="1.795558045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:32.793781889 +0000 UTC m=+1121.942476464" watchObservedRunningTime="2026-11-03 18:56:32.795558045 +0000 UTC m=+1121.944252616"
Nov 03 18:56:32.830793 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:32.830761    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:33.832214 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:33.832178    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 18:56:33.832639 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:33.832236    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 18:56:34.831243 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:34.831208    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 18:56:34.831402 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:34.831266    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 18:56:35.830604 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:35.830575    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 18:56:35.830996 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:35.830625    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 18:56:36.831389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:36.831351    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 18:56:36.831899 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:36.831410    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 18:56:37.831055 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:37.831020    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 18:56:37.831266 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:37.831083    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 18:56:38.996565 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:38.996534    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:38.996978 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:38.996814    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:38.997550 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:38.997534    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 18:56:43.369610 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:43.369571    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"]
Nov 03 18:56:43.370058 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:43.369760    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy" containerID="cri-o://d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0" gracePeriod=30
Nov 03 18:56:47.545761 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:56:47.545692    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:56:48.646401 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.646338    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:56:48.734672 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.734636    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-envoy\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.734846 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.734689    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-data\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.734846 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.734735    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-token\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.734846 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.734787    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-kube-api-access-9mwpm\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-kube-api-access-9mwpm\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.734982 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.734843    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-credential-socket\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.734982 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.734891    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-podinfo\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.734982 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.734977    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-certs\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.735096 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735005    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-data" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:56:48.735096 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735016    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istiod-ca-cert\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.735266 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735098    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-socket\") pod \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\" (UID: \"ab38159d-89dc-48a9-a4e7-9333e3436dcd\") "
Nov 03 18:56:48.735484 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735314    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-certs" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:56:48.735484 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735432    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istiod-ca-cert" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:56:48.735851 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735469    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-socket" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:56:48.735851 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735530    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-credential-socket" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:56:48.735971 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735933    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.735971 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735953    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.736171 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735970    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.736171 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735984    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.736171 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.735998    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.737211 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.737189    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-token" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:56:48.737312 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.737224    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-envoy" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:56:48.737357 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.737307    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-podinfo" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:56:48.737438 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.737421    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-kube-api-access-9mwpm" pod "ab38159d-89dc-48a9-a4e7-9333e3436dcd" (UID: "ab38159d-89dc-48a9-a4e7-9333e3436dcd"). InnerVolumeSpecName "kube-api-access-9mwpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:56:48.774016 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.773985    4489 generic.go:356] "Generic (PLEG): container finished" podID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerID="d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0" exitCode=0
Nov 03 18:56:48.774180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.774038    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"
Nov 03 18:56:48.774180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.774048    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" event={"ID":"ab38159d-89dc-48a9-a4e7-9333e3436dcd","Type":"ContainerDied","Data":"d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0"}
Nov 03 18:56:48.774180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.774099    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx" event={"ID":"ab38159d-89dc-48a9-a4e7-9333e3436dcd","Type":"ContainerDied","Data":"e2e8b57c662b98a332b442b695481583b19927a30de039cb578ff12b278c8cfc"}
Nov 03 18:56:48.774180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.774128    4489 scope.go:122] "RemoveContainer" containerID="d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0"
Nov 03 18:56:48.784303 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.784222    4489 scope.go:122] "RemoveContainer" containerID="d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0"
Nov 03 18:56:48.786351 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:56:48.786325    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0\": container with ID starting with d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0 not found: ID does not exist" containerID="d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0"
Nov 03 18:56:48.786448 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.786360    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0"} err="failed to get container status \"d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0\": rpc error: code = NotFound desc = could not find container \"d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0\": container with ID starting with d497a04734ca2df90d1ddf9cbb9857a205e4bd4881f045c3bad9f3e99955fad0 not found: ID does not exist"
Nov 03 18:56:48.843765 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.843734    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.843765 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.843766    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.843963 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.843779    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mwpm\" (UniqueName: \"kubernetes.io/projected/ab38159d-89dc-48a9-a4e7-9333e3436dcd-kube-api-access-9mwpm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.843963 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.843793    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab38159d-89dc-48a9-a4e7-9333e3436dcd-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:56:48.942282 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.942253    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"]
Nov 03 18:56:48.946269 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.946246    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-bw6rx"]
Nov 03 18:56:48.994997 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:56:48.994968    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" path="/var/lib/kubelet/pods/ab38159d-89dc-48a9-a4e7-9333e3436dcd/volumes"
Nov 03 18:57:17.530459 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:57:17.530416    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:57:18.214876 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.214835    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"]
Nov 03 18:57:18.215440 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.215421    4489 cpu_manager.go:414] "containerMap: removing container" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy"
Nov 03 18:57:18.215557 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.215443    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy"
Nov 03 18:57:18.215614 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.215566    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab38159d-89dc-48a9-a4e7-9333e3436dcd" containerName="istio-proxy"
Nov 03 18:57:18.218694 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.218674    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.273627 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.273598    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-32a59b-openshift-default-dockercfg-rm7v5\""
Nov 03 18:57:18.294267 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294243    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d98c68-36da-4e9a-8f48-79e378d0729c-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294352 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294284    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294352 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294307    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294437 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294364    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294479 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294458    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294514 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294505    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294555 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294522    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294555 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294546    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw752\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-kube-api-access-vw752\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.294688 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.294671    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.329392 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.329347    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"]
Nov 03 18:57:18.395986 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.395956    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d98c68-36da-4e9a-8f48-79e378d0729c-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.395997    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396020    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396038    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396073    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396113    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396166 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396129    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396470 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396186    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw752\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-kube-api-access-vw752\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396470 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396282    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396565    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396636    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396676    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396720    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.396976 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.396775    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d98c68-36da-4e9a-8f48-79e378d0729c-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.398964 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.398939    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.399057 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.398989    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.459708 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.459673    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.459868 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.459739    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw752\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-kube-api-access-vw752\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.530737 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.530657    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:18.959496 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:57:18.959452    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d98c68_36da_4e9a_8f48_79e378d0729c.slice/crio-670e9275a061280d424f8830c2060863bf07318403e6e5e7b9f7adad083a9325 WatchSource:0}: Error finding container 670e9275a061280d424f8830c2060863bf07318403e6e5e7b9f7adad083a9325: Status 404 returned error can't find the container with id 670e9275a061280d424f8830c2060863bf07318403e6e5e7b9f7adad083a9325
Nov 03 18:57:18.961932 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.961902    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:57:18.962003 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.961967    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:57:18.962003 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.961997    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:57:18.962599 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:18.962577    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"]
Nov 03 18:57:19.933606 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:19.933570    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" event={"ID":"05d98c68-36da-4e9a-8f48-79e378d0729c","Type":"ContainerStarted","Data":"45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2"}
Nov 03 18:57:19.934046 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:19.933636    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" event={"ID":"05d98c68-36da-4e9a-8f48-79e378d0729c","Type":"ContainerStarted","Data":"670e9275a061280d424f8830c2060863bf07318403e6e5e7b9f7adad083a9325"}
Nov 03 18:57:20.175085 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:20.175023    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podStartSLOduration=2.175000852 podStartE2EDuration="2.175000852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:20.109112101 +0000 UTC m=+1169.257806684" watchObservedRunningTime="2026-11-03 18:57:20.175000852 +0000 UTC m=+1169.323695423"
Nov 03 18:57:20.531402 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:20.531369    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:20.533044 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:20.533021    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 18:57:20.533169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:20.533065    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 18:57:21.534443 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:21.534402    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 18:57:21.535074 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:21.535041    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 18:57:22.531074 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:22.531045    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 18:57:22.531279 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:22.531098    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 18:57:23.531530 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:23.531496    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 18:57:23.531965 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:23.531557    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 18:57:24.531464 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:24.531432    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 18:57:24.531643 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:24.531483    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 18:57:25.658861 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:25.658828    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:25.659309 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:25.659180    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:25.660077 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:25.660056    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 18:57:40.191533 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.191499    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"]
Nov 03 18:57:40.194959 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.194943    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.251233 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.251203    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"]
Nov 03 18:57:40.268588 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.268550    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tls-certs\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.268732 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.268668    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tmp-dir\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.268786 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.268743    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-model-cache\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.268786 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.268770    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-dshm\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.268873 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.268808    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8f9\" (UniqueName: \"kubernetes.io/projected/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-kube-api-access-zc8f9\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.268932 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.268910    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-home\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.300848 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.300816    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"distinct-w0-9686c3-kserve-self-signed-certs\""
Nov 03 18:57:40.370222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370178    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-model-cache\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370222 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370218    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-dshm\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370445 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370246    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8f9\" (UniqueName: \"kubernetes.io/projected/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-kube-api-access-zc8f9\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370445 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370279    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-home\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370445 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370338    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tls-certs\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370445 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370389    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tmp-dir\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370663 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370645    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-model-cache\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370714 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370696    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-home\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.370763 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.370744    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tmp-dir\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.372659 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.372629    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-dshm\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.372902 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.372884    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tls-certs\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.433478 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.433442    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8f9\" (UniqueName: \"kubernetes.io/projected/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-kube-api-access-zc8f9\") pod \"distinct-w0-9686c3-kserve-5d7f469658-hk24g\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.507016 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.506942    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:40.896374 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:57:40.896346    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5495cb19_e43c_4c3b_9da5_dcdd3f7b87c7.slice/crio-5a594b534ef9775ce2113856ebfcc74f2db592c054dc4d2285d6348721ac0934 WatchSource:0}: Error finding container 5a594b534ef9775ce2113856ebfcc74f2db592c054dc4d2285d6348721ac0934: Status 404 returned error can't find the container with id 5a594b534ef9775ce2113856ebfcc74f2db592c054dc4d2285d6348721ac0934
Nov 03 18:57:40.899699 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:40.899673    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"]
Nov 03 18:57:41.066202 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:41.066166    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" event={"ID":"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7","Type":"ContainerStarted","Data":"de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458"}
Nov 03 18:57:41.066344 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:41.066214    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" event={"ID":"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7","Type":"ContainerStarted","Data":"5a594b534ef9775ce2113856ebfcc74f2db592c054dc4d2285d6348721ac0934"}
Nov 03 18:57:41.188765 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:41.188708    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podStartSLOduration=1.188694243 podStartE2EDuration="1.188694243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:41.186722464 +0000 UTC m=+1190.335417034" watchObservedRunningTime="2026-11-03 18:57:41.188694243 +0000 UTC m=+1190.337388814"
Nov 03 18:57:41.507731 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:41.507646    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:45.898362 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.898329    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"]
Nov 03 18:57:45.901993 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.901969    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:45.942990 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.942963    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-model-cache\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:45.943132 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.943019    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-home\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:45.943279 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.943251    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-dshm\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:45.943333 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.943305    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zmm\" (UniqueName: \"kubernetes.io/projected/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-kube-api-access-q2zmm\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:45.943434 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.943407    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tmp-dir\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:45.943510 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.943497    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tls-certs\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:45.964165 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:45.960335    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"]
Nov 03 18:57:46.007307 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.007276    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"openshift-service-ca.crt\""
Nov 03 18:57:46.007478 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.007321    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"kube-root-ca.crt\""
Nov 03 18:57:46.016440 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.016417    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"default-dockercfg-dbh7c\""
Nov 03 18:57:46.016440 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.016439    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"distinct-w4-2e052a-kserve-self-signed-certs\""
Nov 03 18:57:46.045412 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.045309    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-dshm\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.045619 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.045532    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zmm\" (UniqueName: \"kubernetes.io/projected/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-kube-api-access-q2zmm\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.045696 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.045602    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tmp-dir\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.045800 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.045784    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tls-certs\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.045878 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.045863    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-model-cache\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.046010 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.045993    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-home\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.046663 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.046630    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tmp-dir\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.047970 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.047946    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-model-cache\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.048831 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.048807    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-home\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.050769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.050744    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-dshm\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.050912 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.050883    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tls-certs\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.147201 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.147171    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zmm\" (UniqueName: \"kubernetes.io/projected/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-kube-api-access-q2zmm\") pod \"distinct-w4-2e052a-kserve-5fbf8df994-mg46p\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.215950 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.215922    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:46.588446 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:57:46.588414    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a2a118e_fee9_434d_a571_06d9fb1cbc6b.slice/crio-f06ef22c6c73225585eeefbf2270d36845266dd439b833694efebdb472f4eb4d WatchSource:0}: Error finding container f06ef22c6c73225585eeefbf2270d36845266dd439b833694efebdb472f4eb4d: Status 404 returned error can't find the container with id f06ef22c6c73225585eeefbf2270d36845266dd439b833694efebdb472f4eb4d
Nov 03 18:57:46.592284 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:46.592259    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"]
Nov 03 18:57:47.098907 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:47.098863    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" event={"ID":"7a2a118e-fee9-434d-a571-06d9fb1cbc6b","Type":"ContainerStarted","Data":"4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42"}
Nov 03 18:57:47.099318 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:47.098922    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" event={"ID":"7a2a118e-fee9-434d-a571-06d9fb1cbc6b","Type":"ContainerStarted","Data":"f06ef22c6c73225585eeefbf2270d36845266dd439b833694efebdb472f4eb4d"}
Nov 03 18:57:47.216720 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:47.216694    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:47.223237 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:47.223190    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podStartSLOduration=2.223177179 podStartE2EDuration="2.223177179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:47.222161971 +0000 UTC m=+1196.370856541" watchObservedRunningTime="2026-11-03 18:57:47.223177179 +0000 UTC m=+1196.371871749"
Nov 03 18:57:47.528753 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:57:47.528717    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:57:50.507340 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:50.507312    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:50.519651 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:50.519629    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:51.132199 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:51.132168    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 18:57:56.216697 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:56.216666    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:56.229477 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:56.229452    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:57:57.170156 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:57:57.170114    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 18:58:17.532123 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:58:17.532030    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:58:18.364569 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.364528    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"]
Nov 03 18:58:18.368641 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.368610    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.422973 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.422944    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9686c3-openshift-default-dockercfg-kfklz\""
Nov 03 18:58:18.477310 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.477276    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"]
Nov 03 18:58:18.498361 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498331    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498555 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498397    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498555 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498461    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c59663c-a417-4f26-8ebc-66e4f99e2103-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498555 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498494    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498555 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498533    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498794 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498563    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498794 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498598    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498794 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498631    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.498794 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.498657    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-kube-api-access-p687f\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600346 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600312    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600383    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600429    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600470    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600497    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-kube-api-access-p687f\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600559    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600635    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.600821 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600685    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c59663c-a417-4f26-8ebc-66e4f99e2103-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.601723 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.601698    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.601964 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600894    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.602074 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600981    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.602074 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.601239    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.602074 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.601611    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c59663c-a417-4f26-8ebc-66e4f99e2103-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.602074 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.600754    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.603915 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.603890    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.604179 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.604132    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.666231 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.666204    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.666376 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.666280    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-kube-api-access-p687f\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:18.682730 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:18.682709    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:19.150917 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.150564    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"]
Nov 03 18:58:19.151828 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.151790    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:58:19.151959 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.151875    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:58:19.151959 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.151922    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:58:19.295169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.294702    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" event={"ID":"2c59663c-a417-4f26-8ebc-66e4f99e2103","Type":"ContainerStarted","Data":"3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb"}
Nov 03 18:58:19.295169 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.294773    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" event={"ID":"2c59663c-a417-4f26-8ebc-66e4f99e2103","Type":"ContainerStarted","Data":"305da3d9ac2ccff046901019951e5c6da8a5f6344bb8ae99aeabc73919a2ebe3"}
Nov 03 18:58:19.418434 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.418373    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podStartSLOduration=1.4183536860000001 podStartE2EDuration="1.418353686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:19.418223033 +0000 UTC m=+1228.566917623" watchObservedRunningTime="2026-11-03 18:58:19.418353686 +0000 UTC m=+1228.567048258"
Nov 03 18:58:19.683496 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:19.683396    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:20.685497 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:20.685464    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 18:58:20.685922 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:20.685523    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 18:58:21.683477 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:21.683439    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 18:58:21.683668 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:21.683512    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 18:58:22.683136 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:22.683106    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 18:58:22.683548 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:22.683201    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 18:58:23.683552 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:23.683514    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 18:58:23.684033 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:23.683583    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 18:58:24.684185 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:24.684119    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 18:58:24.684663 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:24.684223    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 18:58:25.683932 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:25.683894    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 18:58:25.684126 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:25.683964    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 18:58:26.687651 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:26.687622    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:26.688092 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:26.687906    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:26.688643 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:26.688627    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 18:58:47.314257 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.314221    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"]
Nov 03 18:58:47.317819 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.317803    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.372470 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.372432    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fba266-openshift-default-dockercfg-dm45z\""
Nov 03 18:58:47.426865 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.426829    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"]
Nov 03 18:58:47.439641 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.439612    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqzt\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-kube-api-access-rpqzt\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.439825 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.439652    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.439825 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.439777    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.439977 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.439878    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.439977 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.439921    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.439977 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.439950    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.440389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.439989    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.440389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.440070    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.440389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.440111    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.529629 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:58:47.529593    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:58:47.541356 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541333    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541501 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541368    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541558 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541496    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqzt\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-kube-api-access-rpqzt\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541558 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541537    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541587    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541660 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541634    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541692    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541731    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541769 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541752    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.541936 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541786    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.542007 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.541978    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.542125 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.542068    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.542125 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.542079    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.542396 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.542377    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.543782 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.543757    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.544063 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.544046    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.606957 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.606879    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.607118 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.607091    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqzt\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-kube-api-access-rpqzt\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:47.638913 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:47.638878    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:48.065456 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.065423    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:58:48.065575 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.065487    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:58:48.065575 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.065518    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 18:58:48.065692 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.065668    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"]
Nov 03 18:58:48.446085 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.446046    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" event={"ID":"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce","Type":"ContainerStarted","Data":"a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431"}
Nov 03 18:58:48.446614 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.446108    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" event={"ID":"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce","Type":"ContainerStarted","Data":"17bd66160078015720baba53197ab2ef3f989be6643883fa6ba7ee5216d2425b"}
Nov 03 18:58:48.572086 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.572032    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podStartSLOduration=1.572018065 podStartE2EDuration="1.572018065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:48.570217462 +0000 UTC m=+1257.718912032" watchObservedRunningTime="2026-11-03 18:58:48.572018065 +0000 UTC m=+1257.720712635"
Nov 03 18:58:48.640004 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:48.639971    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:49.641039 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:49.641004    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 18:58:49.641469 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:49.641056    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 18:58:50.640102 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:50.640068    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 18:58:50.640309 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:50.640130    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 18:58:51.640116 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:51.640081    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 18:58:51.640591 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:51.640172    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 18:58:52.639607 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:52.639567    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 18:58:52.639780 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:52.639628    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 18:58:53.639360 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:53.639322    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 18:58:53.639757 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:53.639400    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 18:58:54.639677 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:54.639640    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 18:58:54.640197 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:54.639732    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 18:58:55.643863 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:55.643833    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:55.644284 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:55.644111    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:58:55.644886 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:58:55.644867    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 18:59:14.005495 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.005462    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"]
Nov 03 18:59:14.009040 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.009021    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.064982 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.064946    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"]
Nov 03 18:59:14.076818 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.076784    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-tmp-dir\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.076980 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.076851    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-dshm\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.077389 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.077350    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-home\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.077517 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.077435    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-model-cache\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.077517 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.077489    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd329e8-de84-4261-b3fe-8e17b51c550a-tls-certs\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.077599 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.077559    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9xz\" (UniqueName: \"kubernetes.io/projected/ddd329e8-de84-4261-b3fe-8e17b51c550a-kube-api-access-pg9xz\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.114221 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.114187    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"distinct-2-w0-9686c3-kserve-self-signed-certs\""
Nov 03 18:59:14.178709 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.178675    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-home\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.178916 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.178731    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-model-cache\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.178916 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.178853    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd329e8-de84-4261-b3fe-8e17b51c550a-tls-certs\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.178916 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.178898    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9xz\" (UniqueName: \"kubernetes.io/projected/ddd329e8-de84-4261-b3fe-8e17b51c550a-kube-api-access-pg9xz\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.179093 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.178967    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-tmp-dir\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.179093 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.179021    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-dshm\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.179218 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.179137    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-model-cache\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.179218 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.179193    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-home\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.179331 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.179313    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-tmp-dir\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.181305 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.181281    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-dshm\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.181535 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.181515    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd329e8-de84-4261-b3fe-8e17b51c550a-tls-certs\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.242779 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.242745    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9xz\" (UniqueName: \"kubernetes.io/projected/ddd329e8-de84-4261-b3fe-8e17b51c550a-kube-api-access-pg9xz\") pod \"distinct-2-w0-9686c3-kserve-689446455c-2sx2f\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") " pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.320880 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.320789    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:14.552385 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.552351    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"]
Nov 03 18:59:14.552625 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.552597    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy" containerID="cri-o://c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd" gracePeriod=30
Nov 03 18:59:14.724258 ip-10-0-143-45 kubenswrapper[4489]: W1103 18:59:14.724217    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddd329e8_de84_4261_b3fe_8e17b51c550a.slice/crio-f60f7baa136e1418354808387ac46d9c79a202c429c87a2e3e9922fbed42a20c WatchSource:0}: Error finding container f60f7baa136e1418354808387ac46d9c79a202c429c87a2e3e9922fbed42a20c: Status 404 returned error can't find the container with id f60f7baa136e1418354808387ac46d9c79a202c429c87a2e3e9922fbed42a20c
Nov 03 18:59:14.727585 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:14.727565    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"]
Nov 03 18:59:15.616470 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:15.616436    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" event={"ID":"ddd329e8-de84-4261-b3fe-8e17b51c550a","Type":"ContainerStarted","Data":"f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8"}
Nov 03 18:59:15.616470 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:15.616485    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" event={"ID":"ddd329e8-de84-4261-b3fe-8e17b51c550a","Type":"ContainerStarted","Data":"f60f7baa136e1418354808387ac46d9c79a202c429c87a2e3e9922fbed42a20c"}
Nov 03 18:59:15.738570 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:15.738514    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podStartSLOduration=2.738498035 podStartE2EDuration="2.738498035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:15.738106367 +0000 UTC m=+1284.886800939" watchObservedRunningTime="2026-11-03 18:59:15.738498035 +0000 UTC m=+1284.887192605"
Nov 03 18:59:16.321013 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:16.320979    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:17.530812 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:59:17.530777    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 18:59:19.816561 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.816497    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:59:19.844328 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844298    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-envoy\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844328 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844334    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-credential-socket\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844544 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844356    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-token\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-token\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844544 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844377    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-podinfo\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844544 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844497    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-socket\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844544 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844533    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-kube-api-access-dbzvt\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-kube-api-access-dbzvt\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844750 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844582    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-data\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844750 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844623    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-certs\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844750 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844741    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istiod-ca-cert\") pod \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\" (UID: \"bde364ef-3c12-4f53-ad45-a1512ef63c7d\") "
Nov 03 18:59:19.844913 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844812    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-socket" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:59:19.844913 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844825    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-data" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:59:19.845057 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.845029    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-certs" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:59:19.845428 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.845407    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.845531 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.845426    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istiod-ca-cert" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:59:19.845531 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.845432    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.845531 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.844899    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-credential-socket" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:59:19.845531 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.845481    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.846801 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.846779    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-token" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:59:19.846892 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.846815    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-podinfo" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:59:19.847246 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.847221    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-envoy" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:59:19.847246 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.847239    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-kube-api-access-dbzvt" pod "bde364ef-3c12-4f53-ad45-a1512ef63c7d" (UID: "bde364ef-3c12-4f53-ad45-a1512ef63c7d"). InnerVolumeSpecName "kube-api-access-dbzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:59:19.946774 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.946740    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.946774 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.946769    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bde364ef-3c12-4f53-ad45-a1512ef63c7d-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.946774 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.946782    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.947010 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.946793    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.947010 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.946803    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbzvt\" (UniqueName: \"kubernetes.io/projected/bde364ef-3c12-4f53-ad45-a1512ef63c7d-kube-api-access-dbzvt\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:19.947010 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:19.946811    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bde364ef-3c12-4f53-ad45-a1512ef63c7d-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:59:20.646927 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.646892    4489 generic.go:356] "Generic (PLEG): container finished" podID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerID="c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd" exitCode=0
Nov 03 18:59:20.647106 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.646944    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"
Nov 03 18:59:20.647106 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.646979    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" event={"ID":"bde364ef-3c12-4f53-ad45-a1512ef63c7d","Type":"ContainerDied","Data":"c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd"}
Nov 03 18:59:20.647106 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.647030    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr" event={"ID":"bde364ef-3c12-4f53-ad45-a1512ef63c7d","Type":"ContainerDied","Data":"71fc96347e4fbc8efa916df3c6b904e6d92db404deaea2f8dbf186f1e89f2668"}
Nov 03 18:59:20.647106 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.647053    4489 scope.go:122] "RemoveContainer" containerID="c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd"
Nov 03 18:59:20.657347 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.657328    4489 scope.go:122] "RemoveContainer" containerID="c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd"
Nov 03 18:59:20.657649 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:59:20.657630    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd\": container with ID starting with c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd not found: ID does not exist" containerID="c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd"
Nov 03 18:59:20.657718 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.657663    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd"} err="failed to get container status \"c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd\": rpc error: code = NotFound desc = could not find container \"c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd\": container with ID starting with c03da7dc318836133db3292d3491ece1d43269027f0ba5418ff8e43fc2bc64bd not found: ID does not exist"
Nov 03 18:59:20.819802 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.819766    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"]
Nov 03 18:59:20.880018 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.874551    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-dffvr"]
Nov 03 18:59:20.994740 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:20.994706    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" path="/var/lib/kubelet/pods/bde364ef-3c12-4f53-ad45-a1512ef63c7d/volumes"
Nov 03 18:59:24.321180 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:24.321133    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:24.333747 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:24.333725    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:24.680058 ip-10-0-143-45 kubenswrapper[4489]: I1103 18:59:24.680033    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 18:59:47.529799 ip-10-0-143-45 kubenswrapper[4489]: E1103 18:59:47.529762    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:00:00.203672 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.203635    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"]
Nov 03 19:00:00.204502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.204478    4489 cpu_manager.go:414] "containerMap: removing container" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy"
Nov 03 19:00:00.204502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.204502    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy"
Nov 03 19:00:00.204701 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.204630    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde364ef-3c12-4f53-ad45-a1512ef63c7d" containerName="istio-proxy"
Nov 03 19:00:00.209279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.209256    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.263604 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.263575    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\""
Nov 03 19:00:00.305991 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.305961    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164588a-4d9b-4575-8cde-b55e5142d5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.306193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.306049    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9n8m\" (UniqueName: \"kubernetes.io/projected/d164588a-4d9b-4575-8cde-b55e5142d5a3-kube-api-access-t9n8m\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.320785 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.320754    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"]
Nov 03 19:00:00.407774 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.407739    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164588a-4d9b-4575-8cde-b55e5142d5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.407921 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.407802    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9n8m\" (UniqueName: \"kubernetes.io/projected/d164588a-4d9b-4575-8cde-b55e5142d5a3-kube-api-access-t9n8m\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.408364 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.408346    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164588a-4d9b-4575-8cde-b55e5142d5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.470971 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.470895    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9n8m\" (UniqueName: \"kubernetes.io/projected/d164588a-4d9b-4575-8cde-b55e5142d5a3-kube-api-access-t9n8m\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.522128 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.522098    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:00.890203 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:00:00.890175    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd164588a_4d9b_4575_8cde_b55e5142d5a3.slice/crio-b54f1bcaf934cc42b444716cb21ebc907a583609672e7b489ff1789914b9870d WatchSource:0}: Error finding container b54f1bcaf934cc42b444716cb21ebc907a583609672e7b489ff1789914b9870d: Status 404 returned error can't find the container with id b54f1bcaf934cc42b444716cb21ebc907a583609672e7b489ff1789914b9870d
Nov 03 19:00:00.892281 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.892261    4489 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:00:00.894114 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:00.894091    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"]
Nov 03 19:00:01.862634 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:01.862590    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" event={"ID":"d164588a-4d9b-4575-8cde-b55e5142d5a3","Type":"ContainerStarted","Data":"b54f1bcaf934cc42b444716cb21ebc907a583609672e7b489ff1789914b9870d"}
Nov 03 19:00:05.488396 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.488315    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"]
Nov 03 19:00:05.491939 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.491921    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.548048 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.548010    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"]
Nov 03 19:00:05.597304 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.597280    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"distinct-2-w1-355e1a-kserve-self-signed-certs\""
Nov 03 19:00:05.666395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.666358    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5vv\" (UniqueName: \"kubernetes.io/projected/7e3760b0-efff-4d4d-96f9-f834a1812dca-kube-api-access-zh5vv\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.666532 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.666512    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-tmp-dir\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.666605 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.666555    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-home\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.666687 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.666665    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-dshm\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.666752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.666698    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-model-cache\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.666821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.666776    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3760b0-efff-4d4d-96f9-f834a1812dca-tls-certs\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.767998 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.767918    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-tmp-dir\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.767998 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.767955    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-home\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.768266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.768083    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-dshm\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.768266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.768124    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-model-cache\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.768266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.768199    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3760b0-efff-4d4d-96f9-f834a1812dca-tls-certs\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.768418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.768351    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5vv\" (UniqueName: \"kubernetes.io/projected/7e3760b0-efff-4d4d-96f9-f834a1812dca-kube-api-access-zh5vv\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.768418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.768368    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-home\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.768418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.768353    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-tmp-dir\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.768562 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.768538    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-model-cache\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.770346 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.770324    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-dshm\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.770706 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.770686    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3760b0-efff-4d4d-96f9-f834a1812dca-tls-certs\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.831862 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.831832    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5vv\" (UniqueName: \"kubernetes.io/projected/7e3760b0-efff-4d4d-96f9-f834a1812dca-kube-api-access-zh5vv\") pod \"distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:05.887086 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.887039    4489 generic.go:356] "Generic (PLEG): container finished" podID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerID="4bd44291d41539bbab04f2926180a37e5724d16e9fd5e1f31905451c7ac7512a" exitCode=60
Nov 03 19:00:05.887281 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.887091    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" event={"ID":"d164588a-4d9b-4575-8cde-b55e5142d5a3","Type":"ContainerDied","Data":"4bd44291d41539bbab04f2926180a37e5724d16e9fd5e1f31905451c7ac7512a"}
Nov 03 19:00:05.887410 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:05.887396    4489 scope.go:122] "RemoveContainer" containerID="4bd44291d41539bbab04f2926180a37e5724d16e9fd5e1f31905451c7ac7512a"
Nov 03 19:00:06.103893 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.103866    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:06.526346 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:00:06.526310    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e3760b0_efff_4d4d_96f9_f834a1812dca.slice/crio-a3ed318473ac64662ffe5847b730c4f36c439199268d8b5a90d12ccf44129460 WatchSource:0}: Error finding container a3ed318473ac64662ffe5847b730c4f36c439199268d8b5a90d12ccf44129460: Status 404 returned error can't find the container with id a3ed318473ac64662ffe5847b730c4f36c439199268d8b5a90d12ccf44129460
Nov 03 19:00:06.531485 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.531454    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"]
Nov 03 19:00:06.896942 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.896849    4489 generic.go:356] "Generic (PLEG): container finished" podID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerID="b642bbef83e7babb8e07ae3b119950dd5b8eac3348e56caf4008f7ae21ef6da8" exitCode=60
Nov 03 19:00:06.897117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.896938    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" event={"ID":"d164588a-4d9b-4575-8cde-b55e5142d5a3","Type":"ContainerDied","Data":"b642bbef83e7babb8e07ae3b119950dd5b8eac3348e56caf4008f7ae21ef6da8"}
Nov 03 19:00:06.897117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.897006    4489 scope.go:122] "RemoveContainer" containerID="4bd44291d41539bbab04f2926180a37e5724d16e9fd5e1f31905451c7ac7512a"
Nov 03 19:00:06.897396 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.897379    4489 scope.go:122] "RemoveContainer" containerID="b642bbef83e7babb8e07ae3b119950dd5b8eac3348e56caf4008f7ae21ef6da8"
Nov 03 19:00:06.897729 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:00:06.897701    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs_odh-ai-gateway-infra(d164588a-4d9b-4575-8cde-b55e5142d5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3"
Nov 03 19:00:06.899497 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.899472    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" event={"ID":"7e3760b0-efff-4d4d-96f9-f834a1812dca","Type":"ContainerStarted","Data":"cf1c3442fdb7d9eff2f20c4c11c8c324b442a646b3c18c435aefdf29ec5b3fdb"}
Nov 03 19:00:06.899569 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:06.899514    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" event={"ID":"7e3760b0-efff-4d4d-96f9-f834a1812dca","Type":"ContainerStarted","Data":"a3ed318473ac64662ffe5847b730c4f36c439199268d8b5a90d12ccf44129460"}
Nov 03 19:00:07.104571 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:07.104535    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:07.137415 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:07.137367    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podStartSLOduration=2.137353109 podStartE2EDuration="2.137353109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:07.136156486 +0000 UTC m=+1336.284851050" watchObservedRunningTime="2026-11-03 19:00:07.137353109 +0000 UTC m=+1336.286047679"
Nov 03 19:00:07.907922 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:07.907894    4489 scope.go:122] "RemoveContainer" containerID="b642bbef83e7babb8e07ae3b119950dd5b8eac3348e56caf4008f7ae21ef6da8"
Nov 03 19:00:07.908399 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:00:07.908230    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs_odh-ai-gateway-infra(d164588a-4d9b-4575-8cde-b55e5142d5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3"
Nov 03 19:00:10.709044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.709017    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"]
Nov 03 19:00:10.712700 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.712682    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.719630 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.719606    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-tmp-dir\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.719846 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.719647    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-dshm\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.719846 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.719824    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126eb0b-1e06-4502-b8c5-c802a034d160-tls-certs\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.720321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.719989    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrn6g\" (UniqueName: \"kubernetes.io/projected/f126eb0b-1e06-4502-b8c5-c802a034d160-kube-api-access-xrn6g\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.720321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.720024    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-model-cache\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.720321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.720052    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-home\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.821655 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.821533    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"distinct-2-w4-2e052a-kserve-self-signed-certs\""
Nov 03 19:00:10.821898 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.821806    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrn6g\" (UniqueName: \"kubernetes.io/projected/f126eb0b-1e06-4502-b8c5-c802a034d160-kube-api-access-xrn6g\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.821898 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.821845    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-model-cache\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.821898 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.821863    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-home\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.822077 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.821901    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-tmp-dir\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.822077 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.821924    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-dshm\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.822077 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.822000    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126eb0b-1e06-4502-b8c5-c802a034d160-tls-certs\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.822996 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.822973    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-home\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.824103 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.823982    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-model-cache\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.824299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.824172    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-tmp-dir\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.826872 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.826843    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-dshm\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.827664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.827643    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126eb0b-1e06-4502-b8c5-c802a034d160-tls-certs\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:10.828189 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.828164    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"]
Nov 03 19:00:10.886932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:10.886888    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrn6g\" (UniqueName: \"kubernetes.io/projected/f126eb0b-1e06-4502-b8c5-c802a034d160-kube-api-access-xrn6g\") pod \"distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") " pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:11.026267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:11.026186    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:11.425996 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:00:11.425966    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf126eb0b_1e06_4502_b8c5_c802a034d160.slice/crio-4723feb34e316cbb5cfec407f77a0d62ae86db92ea492c596c901fb8952a7a53 WatchSource:0}: Error finding container 4723feb34e316cbb5cfec407f77a0d62ae86db92ea492c596c901fb8952a7a53: Status 404 returned error can't find the container with id 4723feb34e316cbb5cfec407f77a0d62ae86db92ea492c596c901fb8952a7a53
Nov 03 19:00:11.430133 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:11.430018    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"]
Nov 03 19:00:11.933448 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:11.933418    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" event={"ID":"f126eb0b-1e06-4502-b8c5-c802a034d160","Type":"ContainerStarted","Data":"7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d"}
Nov 03 19:00:11.933810 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:11.933456    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" event={"ID":"f126eb0b-1e06-4502-b8c5-c802a034d160","Type":"ContainerStarted","Data":"4723feb34e316cbb5cfec407f77a0d62ae86db92ea492c596c901fb8952a7a53"}
Nov 03 19:00:12.026528 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:12.026499    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:12.057011 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:12.056969    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podStartSLOduration=2.056956948 podStartE2EDuration="2.056956948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:12.056575503 +0000 UTC m=+1341.205270076" watchObservedRunningTime="2026-11-03 19:00:12.056956948 +0000 UTC m=+1341.205651519"
Nov 03 19:00:16.104526 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:16.104496    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:16.117323 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:16.117299    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:16.974931 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:16.974906    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:00:17.530190 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:00:17.530153    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:00:19.988781 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:19.988753    4489 scope.go:122] "RemoveContainer" containerID="b642bbef83e7babb8e07ae3b119950dd5b8eac3348e56caf4008f7ae21ef6da8"
Nov 03 19:00:20.988895 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:20.988821    4489 generic.go:356] "Generic (PLEG): container finished" podID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerID="426c395a9e976ae23969feaf1520a6532d227b4ea81a209cb30edc1ab0185aa3" exitCode=60
Nov 03 19:00:20.995016 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:20.994974    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" event={"ID":"d164588a-4d9b-4575-8cde-b55e5142d5a3","Type":"ContainerDied","Data":"426c395a9e976ae23969feaf1520a6532d227b4ea81a209cb30edc1ab0185aa3"}
Nov 03 19:00:20.995173 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:20.995043    4489 scope.go:122] "RemoveContainer" containerID="b642bbef83e7babb8e07ae3b119950dd5b8eac3348e56caf4008f7ae21ef6da8"
Nov 03 19:00:20.995478 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:20.995461    4489 scope.go:122] "RemoveContainer" containerID="426c395a9e976ae23969feaf1520a6532d227b4ea81a209cb30edc1ab0185aa3"
Nov 03 19:00:20.995765 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:00:20.995742    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs_odh-ai-gateway-infra(d164588a-4d9b-4575-8cde-b55e5142d5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3"
Nov 03 19:00:21.026804 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.026782    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:21.039037 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.039017    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:21.759763 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.759728    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"]
Nov 03 19:00:21.764460 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.764442    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.821568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.821538    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"]
Nov 03 19:00:21.842711 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.842688    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-home\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.842840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.842718    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5dw\" (UniqueName: \"kubernetes.io/projected/9c2859a5-32ed-4267-8838-3430edc17c3c-kube-api-access-qj5dw\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.842896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.842875    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2859a5-32ed-4267-8838-3430edc17c3c-tls-certs\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.842932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.842919    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-dshm\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.842993 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.842981    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-tmp-dir\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.843030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.843017    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-model-cache\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.869761 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.869738    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"distinct-2-w5-3a7056-kserve-self-signed-certs\""
Nov 03 19:00:21.944169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944112    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2859a5-32ed-4267-8838-3430edc17c3c-tls-certs\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944341 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944172    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-dshm\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944341 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944209    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-tmp-dir\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944341 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944229    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-model-cache\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944341 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944318    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-home\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944488 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944343    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5dw\" (UniqueName: \"kubernetes.io/projected/9c2859a5-32ed-4267-8838-3430edc17c3c-kube-api-access-qj5dw\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944692 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944666    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-tmp-dir\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944692 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944685    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-model-cache\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.944836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.944724    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-home\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.946688 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.946666    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-dshm\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:21.946969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:21.946949    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2859a5-32ed-4267-8838-3430edc17c3c-tls-certs\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:22.007462 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.007434    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:00:22.009691 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.009670    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5dw\" (UniqueName: \"kubernetes.io/projected/9c2859a5-32ed-4267-8838-3430edc17c3c-kube-api-access-qj5dw\") pod \"distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") " pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:22.077329 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.077267    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:22.127282 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.127251    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"]
Nov 03 19:00:22.316565 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.316536    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:22.451825 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.451798    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d164588a-4d9b-4575-8cde-b55e5142d5a3-kube-api-access-t9n8m\" (UniqueName: \"kubernetes.io/projected/d164588a-4d9b-4575-8cde-b55e5142d5a3-kube-api-access-t9n8m\") pod \"d164588a-4d9b-4575-8cde-b55e5142d5a3\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") "
Nov 03 19:00:22.451998 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.451844    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d164588a-4d9b-4575-8cde-b55e5142d5a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164588a-4d9b-4575-8cde-b55e5142d5a3-maas-api-service-ca\") pod \"d164588a-4d9b-4575-8cde-b55e5142d5a3\" (UID: \"d164588a-4d9b-4575-8cde-b55e5142d5a3\") "
Nov 03 19:00:22.452292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.452267    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d164588a-4d9b-4575-8cde-b55e5142d5a3-maas-api-service-ca" pod "d164588a-4d9b-4575-8cde-b55e5142d5a3" (UID: "d164588a-4d9b-4575-8cde-b55e5142d5a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:22.454105 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.454082    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d164588a-4d9b-4575-8cde-b55e5142d5a3-kube-api-access-t9n8m" pod "d164588a-4d9b-4575-8cde-b55e5142d5a3" (UID: "d164588a-4d9b-4575-8cde-b55e5142d5a3"). InnerVolumeSpecName "kube-api-access-t9n8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:22.490771 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:00:22.490738    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c2859a5_32ed_4267_8838_3430edc17c3c.slice/crio-257cee8de4152d35e7be731e8619b402c044e22177a2c6fa626e7ca2eef126d6 WatchSource:0}: Error finding container 257cee8de4152d35e7be731e8619b402c044e22177a2c6fa626e7ca2eef126d6: Status 404 returned error can't find the container with id 257cee8de4152d35e7be731e8619b402c044e22177a2c6fa626e7ca2eef126d6
Nov 03 19:00:22.495116 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.495088    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"]
Nov 03 19:00:22.553548 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.553526    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9n8m\" (UniqueName: \"kubernetes.io/projected/d164588a-4d9b-4575-8cde-b55e5142d5a3-kube-api-access-t9n8m\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:22.553640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:22.553550    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164588a-4d9b-4575-8cde-b55e5142d5a3-maas-api-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:23.004432 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.004394    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" event={"ID":"9c2859a5-32ed-4267-8838-3430edc17c3c","Type":"ContainerStarted","Data":"d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7"}
Nov 03 19:00:23.004632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.004444    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" event={"ID":"9c2859a5-32ed-4267-8838-3430edc17c3c","Type":"ContainerStarted","Data":"257cee8de4152d35e7be731e8619b402c044e22177a2c6fa626e7ca2eef126d6"}
Nov 03 19:00:23.006015 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.005989    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"
Nov 03 19:00:23.006189 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.006062    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs" event={"ID":"d164588a-4d9b-4575-8cde-b55e5142d5a3","Type":"ContainerDied","Data":"b54f1bcaf934cc42b444716cb21ebc907a583609672e7b489ff1789914b9870d"}
Nov 03 19:00:23.006189 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.006112    4489 scope.go:122] "RemoveContainer" containerID="426c395a9e976ae23969feaf1520a6532d227b4ea81a209cb30edc1ab0185aa3"
Nov 03 19:00:23.078003 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.077975    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:23.244583 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.244488    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podStartSLOduration=2.244471649 podStartE2EDuration="2.244471649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:23.129288027 +0000 UTC m=+1352.277982599" watchObservedRunningTime="2026-11-03 19:00:23.244471649 +0000 UTC m=+1352.393166218"
Nov 03 19:00:23.245708 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.245574    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"]
Nov 03 19:00:23.254336 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:23.254312    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895540-zncfs"]
Nov 03 19:00:24.995085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:24.995041    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" path="/var/lib/kubelet/pods/d164588a-4d9b-4575-8cde-b55e5142d5a3/volumes"
Nov 03 19:00:32.077803 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:32.077774    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:32.091897 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:32.091874    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:33.076646 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:33.076620    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:00:47.532782 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:00:47.532744    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:00:48.463444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.463412    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"]
Nov 03 19:00:48.463932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.463913    4489 cpu_manager.go:414] "containerMap: removing container" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.464024 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.463933    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.464024 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.463968    4489 cpu_manager.go:414] "containerMap: removing container" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.464024 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.463977    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.464024 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.463993    4489 cpu_manager.go:414] "containerMap: removing container" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.464024 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.464001    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.464302 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.464130    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.464302 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.464163    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.467444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.467424    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.576505 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.576470    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"]
Nov 03 19:00:48.634751 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.634716    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b67856f-b54a-4f38-a021-98e6d710f254-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.634919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.634847    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.634919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.634899    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.634919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.634916    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.635085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.634952    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.635085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.635000    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b67856f-b54a-4f38-a021-98e6d710f254-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.635227 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.635207    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.635286 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.635240    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.635371 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.635356    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwh9j\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-kube-api-access-jwh9j\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.736747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.736664    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwh9j\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-kube-api-access-jwh9j\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.736747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.736708    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b67856f-b54a-4f38-a021-98e6d710f254-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.736956 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.736799    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.736956 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.736852    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.736956 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.736875    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.736956 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.736911    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.737127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.737037    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b67856f-b54a-4f38-a021-98e6d710f254-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.737127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.737111    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.737273 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.737134    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.737332 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.737312    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.737430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.737399    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.737489 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.737419    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.737604 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.737583    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.738064 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.738038    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b67856f-b54a-4f38-a021-98e6d710f254-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.739404 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.739385    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.739527 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.739510    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b67856f-b54a-4f38-a021-98e6d710f254-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.800440 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.800403    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.800583 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.800519    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwh9j\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-kube-api-access-jwh9j\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:48.877186 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.877053    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"]
Nov 03 19:00:48.877769 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.877749    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="d164588a-4d9b-4575-8cde-b55e5142d5a3" containerName="cleanup"
Nov 03 19:00:48.882240 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.882223    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:48.927352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.927321    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"]
Nov 03 19:00:48.936005 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.935983    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:48.942324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:48.942294    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"]
Nov 03 19:00:49.007826 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.007738    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"]
Nov 03 19:00:49.041415 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041384    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.041415 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041416    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.041640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041502    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.041640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041545    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.041640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041562    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2lt\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-kube-api-access-9s2lt\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.041640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041621    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.041818 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041691    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.041818 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041766    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.041818 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041804    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.041949 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041831    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.041949 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041849    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.041949 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041879    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.042067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.041970    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.042067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.042015    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.042067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.042031    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhdb\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-kube-api-access-bbhdb\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.042229 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.042076    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.042229 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.042111    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.042229 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.042203    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.083345 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.083322    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:49.142982 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.142944    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143081 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.142999    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143120 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143075    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143179 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143119    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143219 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143202    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2lt\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-kube-api-access-9s2lt\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143256 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143233    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143331 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143314    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143362 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143350    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143381    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143445 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143431    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143479 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143461    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143522 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143508    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143589 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143576    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143678 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143664    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143735 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143695    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhdb\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-kube-api-access-bbhdb\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143782 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143734    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.143782 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143768    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.143885 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.143844    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.144361 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.144340    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.145179 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.145000    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.145179 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.145049    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.146199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.146173    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.146320 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.146293    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.147130 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.147104    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.147406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.147386    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.147474 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.147384    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.147552 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.147533    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.147759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.147742    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.159719 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.159692    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.159837 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.159765    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.159969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.159946    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.161543 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.161514    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.219428 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.219398    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhdb\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-kube-api-access-bbhdb\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.232729 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.232697    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.232905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.232886    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2lt\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-kube-api-access-9s2lt\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.235097 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.235079    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.253335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.253295    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:49.496327 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.496293    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:49.506636 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:00:49.506610    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b67856f_b54a_4f38_a021_98e6d710f254.slice/crio-9d6d1b8ade1a86a09104d45a62a14ee489cc26415bd98d5edb5aa97d8cc0fed4 WatchSource:0}: Error finding container 9d6d1b8ade1a86a09104d45a62a14ee489cc26415bd98d5edb5aa97d8cc0fed4: Status 404 returned error can't find the container with id 9d6d1b8ade1a86a09104d45a62a14ee489cc26415bd98d5edb5aa97d8cc0fed4
Nov 03 19:00:49.507359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.507336    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"]
Nov 03 19:00:49.509731 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.509694    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.509807 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.509786    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.509854 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.509840    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.637218 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:00:49.637189    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe4fc5b_6e44_4231_8ebd_6b351867d0cf.slice/crio-f0704a92b1d8f047a4de974bec87c0ece343a74b4a6be8c6780410e0d862fc1e WatchSource:0}: Error finding container f0704a92b1d8f047a4de974bec87c0ece343a74b4a6be8c6780410e0d862fc1e: Status 404 returned error can't find the container with id f0704a92b1d8f047a4de974bec87c0ece343a74b4a6be8c6780410e0d862fc1e
Nov 03 19:00:49.640401 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.640363    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.640527 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.640448    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.640527 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.640487    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.641245 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.641214    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"]
Nov 03 19:00:49.879558 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:00:49.879521    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b63ba25_64d3_4ae9_8e7c_d7572376d097.slice/crio-d2e305a11b668104d411a62781758e8a6855255a1b49c0b3dc37fb4f34a281c1 WatchSource:0}: Error finding container d2e305a11b668104d411a62781758e8a6855255a1b49c0b3dc37fb4f34a281c1: Status 404 returned error can't find the container with id d2e305a11b668104d411a62781758e8a6855255a1b49c0b3dc37fb4f34a281c1
Nov 03 19:00:49.882762 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.882732    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"]
Nov 03 19:00:49.882918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.882888    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.882984 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.882959    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:49.883036 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:49.883002    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:00:50.186981 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.186942    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" event={"ID":"1b67856f-b54a-4f38-a021-98e6d710f254","Type":"ContainerStarted","Data":"2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702"}
Nov 03 19:00:50.187203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.187001    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" event={"ID":"1b67856f-b54a-4f38-a021-98e6d710f254","Type":"ContainerStarted","Data":"9d6d1b8ade1a86a09104d45a62a14ee489cc26415bd98d5edb5aa97d8cc0fed4"}
Nov 03 19:00:50.188918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.188892    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" event={"ID":"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf","Type":"ContainerStarted","Data":"65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea"}
Nov 03 19:00:50.189053 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.188940    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" event={"ID":"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf","Type":"ContainerStarted","Data":"f0704a92b1d8f047a4de974bec87c0ece343a74b4a6be8c6780410e0d862fc1e"}
Nov 03 19:00:50.190600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.190576    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" event={"ID":"4b63ba25-64d3-4ae9-8e7c-d7572376d097","Type":"ContainerStarted","Data":"0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa"}
Nov 03 19:00:50.190707 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.190615    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" event={"ID":"4b63ba25-64d3-4ae9-8e7c-d7572376d097","Type":"ContainerStarted","Data":"d2e305a11b668104d411a62781758e8a6855255a1b49c0b3dc37fb4f34a281c1"}
Nov 03 19:00:50.253981 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.253948    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:50.256101 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.256062    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:00:50.256266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.256119    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:00:50.312325 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.312264    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podStartSLOduration=2.312244569 podStartE2EDuration="2.312244569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.31052169 +0000 UTC m=+1379.459216261" watchObservedRunningTime="2026-11-03 19:00:50.312244569 +0000 UTC m=+1379.460939141"
Nov 03 19:00:50.497194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.497090    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:50.499208 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.499183    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:00:50.499354 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.499233    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:00:50.567169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:50.563807    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podStartSLOduration=2.563784439 podStartE2EDuration="2.563784439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.438619297 +0000 UTC m=+1379.587313949" watchObservedRunningTime="2026-11-03 19:00:50.563784439 +0000 UTC m=+1379.712479013"
Nov 03 19:00:51.100177 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:51.091623    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:51.125170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:51.117290    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:00:51.125170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:51.117374    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:00:51.316018 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:51.315968    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:00:51.316379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:51.316344    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:00:51.518168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:51.517546    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:00:51.518168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:51.517610    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:00:52.086696 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:52.086658    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:00:52.087621 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:52.087587    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:00:52.280170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:52.279799    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:00:52.280170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:52.279855    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:00:52.498191 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:52.497462    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:00:52.498191 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:52.497524    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:00:53.088169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:53.087510    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:00:53.088169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:53.087572    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:00:53.268171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:53.267522    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:00:53.268171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:53.267587    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:00:53.498532 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:53.498495    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:00:53.499172 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:53.499110    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:00:54.088288 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:54.088252    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:00:54.088588 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:54.088555    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:00:54.325171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:54.324482    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:00:54.325171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:54.324542    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:00:54.512167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:54.497973    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:00:54.512167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:54.498031    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:00:55.085169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:55.084854    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:00:55.085169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:55.084926    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:00:55.279169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:55.278277    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:00:55.279169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:55.278361    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:00:55.504166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:55.503495    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:00:55.504166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:55.503556    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:00:56.086167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:56.085570    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:00:56.086167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:56.085633    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:00:56.254485 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:56.254437    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:00:56.254681 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:56.254500    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 19:00:56.496849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:56.496809    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 19:00:56.497067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:56.496877    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 19:00:57.648365 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.648268    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:57.648886 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.648691    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:57.649519 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.649499    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:00:57.653305 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.653288    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:57.653580 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.653564    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:57.654480 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.654460    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:00:57.745587 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.745551    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:57.746134 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.746017    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:57.747007 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.746988    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:00:57.826408 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:00:57.826358    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podStartSLOduration=9.826343349 podStartE2EDuration="9.826343349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.567276636 +0000 UTC m=+1379.715971207" watchObservedRunningTime="2026-11-03 19:00:57.826343349 +0000 UTC m=+1386.975037919"
Nov 03 19:01:17.529484 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:01:17.529404    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:01:20.066912 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.066881    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"]
Nov 03 19:01:20.070872 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.070852    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.121751 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.121719    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.121951 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.121879    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdh94\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-kube-api-access-rdh94\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.121951 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.121917    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.122118 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.122091    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.122251 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.122224    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.122386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.122268    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.122447 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.122396    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.122501 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.122456    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.122501 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.122481    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.154012 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.153977    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"]
Nov 03 19:01:20.223604 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223564    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.223604 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223609    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.223861 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223626    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.223861 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223813    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.223861 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223845    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224029 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223883    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224029 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223940    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224029 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.223968    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224235 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.224066    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdh94\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-kube-api-access-rdh94\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224235 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.224107    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224235 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.224131    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.224275    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224606 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.224434    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.224606 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.224591    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.226072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.226051    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.226244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.226225    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.287286 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.287254    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.287403 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.287321    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdh94\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-kube-api-access-rdh94\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.371980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.371911    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"]
Nov 03 19:01:20.377254 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.377121    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.384689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.384664    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:20.425728 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.425688    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.425728 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.425725    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.425976 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.425754    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.425976 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.425931    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436255e1-65db-44db-b906-814953040195-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.426089 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.426042    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.426185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.426155    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.426248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.426218    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.428184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.427647    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2cl\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-kube-api-access-2w2cl\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.428184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.427690    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436255e1-65db-44db-b906-814953040195-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.444335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.435814    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"]
Nov 03 19:01:20.529225 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529194    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529381 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529251    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529381 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529294    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529475 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529441    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2cl\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-kube-api-access-2w2cl\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529527 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529473    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436255e1-65db-44db-b906-814953040195-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529582 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529557    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529588    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529604    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529620    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529625    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529683    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436255e1-65db-44db-b906-814953040195-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529843    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.529913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.529885    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.530185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.530132    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436255e1-65db-44db-b906-814953040195-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.531891 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.531870    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.532180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.532161    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436255e1-65db-44db-b906-814953040195-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.592265 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.592238    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.592384 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.592367    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2cl\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-kube-api-access-2w2cl\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5\" (UID: \"436255e1-65db-44db-b906-814953040195\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.693304 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.693273    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:20.808388 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:01:20.808338    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb54b5f0_0fa8_4f5b_9255_ccc89e323396.slice/crio-3957cb4ad100b74315f6fccbd42d85bd872f97c56e8c1a0c519ade29bf0559c8 WatchSource:0}: Error finding container 3957cb4ad100b74315f6fccbd42d85bd872f97c56e8c1a0c519ade29bf0559c8: Status 404 returned error can't find the container with id 3957cb4ad100b74315f6fccbd42d85bd872f97c56e8c1a0c519ade29bf0559c8
Nov 03 19:01:20.811742 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.811714    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"]
Nov 03 19:01:20.812242 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.812201    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:01:20.812365 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.812287    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:01:20.812365 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:20.812338    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:01:21.067985 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:01:21.067948    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436255e1_65db_44db_b906_814953040195.slice/crio-a7bcf47e216771370c15688a7c4a80102c75cbfee5340e0e4551c3e5c4e59f7f WatchSource:0}: Error finding container a7bcf47e216771370c15688a7c4a80102c75cbfee5340e0e4551c3e5c4e59f7f: Status 404 returned error can't find the container with id a7bcf47e216771370c15688a7c4a80102c75cbfee5340e0e4551c3e5c4e59f7f
Nov 03 19:01:21.070949 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.070917    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:01:21.071045 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.070984    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:01:21.071045 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.071013    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:01:21.072576 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.072547    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"]
Nov 03 19:01:21.497191 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.497120    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" event={"ID":"436255e1-65db-44db-b906-814953040195","Type":"ContainerStarted","Data":"cf7c939d59d2266d729de4725c74bea4ec3ceb58e20eea860d4630b8eab2830a"}
Nov 03 19:01:21.497365 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.497223    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" event={"ID":"436255e1-65db-44db-b906-814953040195","Type":"ContainerStarted","Data":"a7bcf47e216771370c15688a7c4a80102c75cbfee5340e0e4551c3e5c4e59f7f"}
Nov 03 19:01:21.498974 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.498946    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" event={"ID":"fb54b5f0-0fa8-4f5b-9255-ccc89e323396","Type":"ContainerStarted","Data":"a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec"}
Nov 03 19:01:21.499095 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.498997    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" event={"ID":"fb54b5f0-0fa8-4f5b-9255-ccc89e323396","Type":"ContainerStarted","Data":"3957cb4ad100b74315f6fccbd42d85bd872f97c56e8c1a0c519ade29bf0559c8"}
Nov 03 19:01:21.625207 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.625134    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podStartSLOduration=1.6251186340000001 podStartE2EDuration="1.625118634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:21.623728806 +0000 UTC m=+1410.772423395" watchObservedRunningTime="2026-11-03 19:01:21.625118634 +0000 UTC m=+1410.773813204"
Nov 03 19:01:21.695169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.694248    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:21.770254 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:21.765254    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podStartSLOduration=1.765234755 podStartE2EDuration="1.765234755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:21.764454492 +0000 UTC m=+1410.913149064" watchObservedRunningTime="2026-11-03 19:01:21.765234755 +0000 UTC m=+1410.913929350"
Nov 03 19:01:22.388171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:22.387497    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:22.409714 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:22.400323    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:01:22.409714 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:22.400380    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:01:22.696980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:22.696941    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:01:22.697293 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:22.697256    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:01:22.972227 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:22.972008    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:01:23.004173 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.000620    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.077167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.076232    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.099170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.094226    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.099170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.094609    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.134167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.133228    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:01:23.164273 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.164232    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.280338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.280233    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:01:23.305277 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.305230    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.326265 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.326223    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.417168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.411378    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:01:23.417168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.411434    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:01:23.540230 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.531348    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.700377 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.673251    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.700377 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.695507    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:01:23.700377 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.695564    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:01:23.766169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.765234    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.854224 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.853767    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.986168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.984362    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:23.986168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:23.985070    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:01:24.067167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.058307    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.067167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.058715    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:01:24.082167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.078248    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.082167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.078681    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:01:24.107226 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.096231    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.107226 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.096632    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:01:24.149167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.134400    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.149167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.134764    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:01:24.174168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.167232    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.174168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.173699    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:01:24.301975 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.301934    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.304044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.304021    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:01:24.341169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.337233    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.341169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.337652    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:01:24.545509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.545474    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:01:24.626188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.625910    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:01:24.628167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.628108    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Nov 03 19:01:24.628942 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.628817    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 19:01:24.633349 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.633332    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:01:24.649167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.634771    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:01:24.670538 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.670454    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:01:24.674416 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.674386    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:01:24.676239 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.675463    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 19:01:24.725476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.725445    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:01:24.725741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.725716    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:01:24.818960 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.818862    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 19:01:24.835654 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.835624    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:01:24.915169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.896944    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:01:24.915169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.897019    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:01:24.915169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.897498    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:01:24.915169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.898484    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 19:01:24.916168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:24.915634    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:01:25.409545 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:25.409502    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:01:25.409750 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:25.409583    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:01:25.698728 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:25.698639    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:01:25.700764 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:25.700260    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:01:26.404196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:26.391527    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:01:26.404196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:26.391599    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:01:26.693726 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:26.693691    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:01:26.694072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:26.694026    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:01:27.402857 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:27.402815    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:01:27.403417 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:27.402880    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 19:01:27.694013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:27.693951    4489 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:01:27.694254 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:27.694012    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:01:28.797421 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:28.796847    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:28.797931 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:28.797783    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:28.798765 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:28.798740    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:01:29.038435 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:29.038404    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:29.038739 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:29.038718    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:29.039754 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:29.039735    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:01:47.530771 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:01:47.530730    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:01:52.098279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.098245    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-c401af"]
Nov 03 19:01:52.102092 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.102073    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:01:52.155055 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.155027    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-c401af"]
Nov 03 19:01:52.203715 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.203690    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zmjtl\""
Nov 03 19:01:52.215698 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.215671    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8nd\" (UniqueName: \"kubernetes.io/projected/bcac5eb3-05bc-44bf-a88f-e50441341c1b-kube-api-access-wh8nd\") pod \"test-curl-7315-c401af\" (UID: \"bcac5eb3-05bc-44bf-a88f-e50441341c1b\") " pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:01:52.316975 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.316941    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8nd\" (UniqueName: \"kubernetes.io/projected/bcac5eb3-05bc-44bf-a88f-e50441341c1b-kube-api-access-wh8nd\") pod \"test-curl-7315-c401af\" (UID: \"bcac5eb3-05bc-44bf-a88f-e50441341c1b\") " pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:01:52.380759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.380683    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8nd\" (UniqueName: \"kubernetes.io/projected/bcac5eb3-05bc-44bf-a88f-e50441341c1b-kube-api-access-wh8nd\") pod \"test-curl-7315-c401af\" (UID: \"bcac5eb3-05bc-44bf-a88f-e50441341c1b\") " pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:01:52.414481 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.414434    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:01:52.787365 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:01:52.787328    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcac5eb3_05bc_44bf_a88f_e50441341c1b.slice/crio-2da0b29b56905d3b16eb5bd5e2be7d2452124708d07248c3451fa997c5235b1d WatchSource:0}: Error finding container 2da0b29b56905d3b16eb5bd5e2be7d2452124708d07248c3451fa997c5235b1d: Status 404 returned error can't find the container with id 2da0b29b56905d3b16eb5bd5e2be7d2452124708d07248c3451fa997c5235b1d
Nov 03 19:01:52.789109 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.789083    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-c401af"]
Nov 03 19:01:52.978950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:52.978911    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-c401af" event={"ID":"bcac5eb3-05bc-44bf-a88f-e50441341c1b","Type":"ContainerStarted","Data":"2da0b29b56905d3b16eb5bd5e2be7d2452124708d07248c3451fa997c5235b1d"}
Nov 03 19:01:55.996894 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:55.996804    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-c401af" event={"ID":"bcac5eb3-05bc-44bf-a88f-e50441341c1b","Type":"ContainerStarted","Data":"b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c"}
Nov 03 19:01:56.116035 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:56.115975    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-c401af" podStartSLOduration=1.184376803 podStartE2EDuration="4.115961019s" totalImagesPullingTime="2.931584216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:56.114791278 +0000 UTC m=+1445.263485852" watchObservedRunningTime="2026-11-03 19:01:56.115961019 +0000 UTC m=+1445.264655589"
Nov 03 19:01:57.641827 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.641794    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-c401af"]
Nov 03 19:01:57.645790 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.645759    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-c401af"]
Nov 03 19:01:57.802650 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.802617    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-714cf1"]
Nov 03 19:01:57.803095 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.803081    4489 cpu_manager.go:414] "containerMap: removing container" podUID="bcac5eb3-05bc-44bf-a88f-e50441341c1b" containerName="test-curl-7315-c401af"
Nov 03 19:01:57.803176 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.803096    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcac5eb3-05bc-44bf-a88f-e50441341c1b" containerName="test-curl-7315-c401af"
Nov 03 19:01:57.803235 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.803217    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcac5eb3-05bc-44bf-a88f-e50441341c1b" containerName="test-curl-7315-c401af"
Nov 03 19:01:57.806551 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.806528    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:01:57.859253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.859224    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-714cf1"]
Nov 03 19:01:57.982605 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:57.982569    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vw7\" (UniqueName: \"kubernetes.io/projected/ccd0df80-fd02-4b89-aa5f-11eb2559e390-kube-api-access-w2vw7\") pod \"test-curl-7315-714cf1\" (UID: \"ccd0df80-fd02-4b89-aa5f-11eb2559e390\") " pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:01:58.018384 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:58.018317    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-c401af" podUID="bcac5eb3-05bc-44bf-a88f-e50441341c1b" containerName="test-curl-7315-c401af" containerID="cri-o://b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c" gracePeriod=2
Nov 03 19:01:58.071739 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:01:58.071699    4489 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-c401af\" is forbidden: User \"system:node:ip-10-0-143-45.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-45.us-west-2.compute.internal' and this object" podUID="bcac5eb3-05bc-44bf-a88f-e50441341c1b" pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:01:58.084108 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:58.084076    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vw7\" (UniqueName: \"kubernetes.io/projected/ccd0df80-fd02-4b89-aa5f-11eb2559e390-kube-api-access-w2vw7\") pod \"test-curl-7315-714cf1\" (UID: \"ccd0df80-fd02-4b89-aa5f-11eb2559e390\") " pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:01:58.147805 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:58.147772    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vw7\" (UniqueName: \"kubernetes.io/projected/ccd0df80-fd02-4b89-aa5f-11eb2559e390-kube-api-access-w2vw7\") pod \"test-curl-7315-714cf1\" (UID: \"ccd0df80-fd02-4b89-aa5f-11eb2559e390\") " pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:01:58.417819 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:58.417785    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:01:58.805165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:58.798395    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-714cf1"]
Nov 03 19:01:59.025753 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:59.025719    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-714cf1" event={"ID":"ccd0df80-fd02-4b89-aa5f-11eb2559e390","Type":"ContainerStarted","Data":"465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8"}
Nov 03 19:01:59.025925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:59.025767    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-714cf1" event={"ID":"ccd0df80-fd02-4b89-aa5f-11eb2559e390","Type":"ContainerStarted","Data":"6202bf839ab8b3bae94d2103278352858577be378a4a0dfffdd9c63acbd3c333"}
Nov 03 19:01:59.146427 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:01:59.146299    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-714cf1" podStartSLOduration=2.146281056 podStartE2EDuration="2.146281056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:59.143457239 +0000 UTC m=+1448.292151812" watchObservedRunningTime="2026-11-03 19:01:59.146281056 +0000 UTC m=+1448.294975632"
Nov 03 19:02:00.803205 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:00.800239    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:02:00.827026 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:00.826233    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:00.858469 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:00.857644    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:00.906546 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:00.906518    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:02:00.932180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:00.928429    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:02:01.078260 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.078225    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcac5eb3-05bc-44bf-a88f-e50441341c1b-kube-api-access-wh8nd\" (UniqueName: \"kubernetes.io/projected/bcac5eb3-05bc-44bf-a88f-e50441341c1b-kube-api-access-wh8nd\") pod \"bcac5eb3-05bc-44bf-a88f-e50441341c1b\" (UID: \"bcac5eb3-05bc-44bf-a88f-e50441341c1b\") "
Nov 03 19:02:01.111809 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.111400    4489 generic.go:356] "Generic (PLEG): container finished" podID="bcac5eb3-05bc-44bf-a88f-e50441341c1b" containerID="b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c" exitCode=137
Nov 03 19:02:01.111809 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.111537    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c401af"
Nov 03 19:02:01.133248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.133199    4489 scope.go:122] "RemoveContainer" containerID="b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c"
Nov 03 19:02:01.173689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.173649    4489 scope.go:122] "RemoveContainer" containerID="b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c"
Nov 03 19:02:01.175419 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:02:01.175395    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c\": container with ID starting with b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c not found: ID does not exist" containerID="b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c"
Nov 03 19:02:01.175594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.175567    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c"} err="failed to get container status \"b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c\": rpc error: code = NotFound desc = could not find container \"b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c\": container with ID starting with b7d0c64eefe04aec68c6db67961e804012baf39a60ffc75a34f848d22490e27c not found: ID does not exist"
Nov 03 19:02:01.258939 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.258905    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcac5eb3-05bc-44bf-a88f-e50441341c1b-kube-api-access-wh8nd" pod "bcac5eb3-05bc-44bf-a88f-e50441341c1b" (UID: "bcac5eb3-05bc-44bf-a88f-e50441341c1b"). InnerVolumeSpecName "kube-api-access-wh8nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:01.289950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.288007    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh8nd\" (UniqueName: \"kubernetes.io/projected/bcac5eb3-05bc-44bf-a88f-e50441341c1b-kube-api-access-wh8nd\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:01.511527 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.511477    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:01.649166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.645623    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-714cf1"]
Nov 03 19:02:01.649166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.645797    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-714cf1" podUID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" containerName="test-curl-7315-714cf1" containerID="cri-o://465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8" gracePeriod=2
Nov 03 19:02:01.685180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.681195    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-714cf1"]
Nov 03 19:02:01.737170 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:02:01.727398    4489 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-714cf1\" is forbidden: User \"system:node:ip-10-0-143-45.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-45.us-west-2.compute.internal' and this object" podUID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:02:01.930177 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.923217    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:02:01.930177 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:01.924744    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Nov 03 19:02:02.995490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:02.995455    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcac5eb3-05bc-44bf-a88f-e50441341c1b" path="/var/lib/kubelet/pods/bcac5eb3-05bc-44bf-a88f-e50441341c1b/volumes"
Nov 03 19:02:03.061720 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.061685    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"]
Nov 03 19:02:03.062279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.062262    4489 cpu_manager.go:414] "containerMap: removing container" podUID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" containerName="test-curl-7315-714cf1"
Nov 03 19:02:03.062279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.062279    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" containerName="test-curl-7315-714cf1"
Nov 03 19:02:03.062415 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.062379    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" containerName="test-curl-7315-714cf1"
Nov 03 19:02:03.067211 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.067192    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.125391 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.125361    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"]
Nov 03 19:02:03.172188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172080    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-credential-socket\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172368 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172227    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-socket\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172368 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172297    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-token\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172372    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbtrk\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-kube-api-access-cbtrk\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172417    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1477b2-7e81-4df4-b942-ca04c61934aa-istiod-ca-cert\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172456    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-envoy\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172651 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172602    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-certs\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172651 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172624    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-podinfo\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.172651 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.172646    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-data\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.173426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.173396    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-90774e61-openshift-default-dockercfg-hmbc9\""
Nov 03 19:02:03.273852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.273765    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-envoy\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.273889    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-certs\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.273911    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-podinfo\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.273929    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-data\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.273988    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-credential-socket\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274257 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274053    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-socket\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274257 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274113    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-token\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274257 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274170    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbtrk\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-kube-api-access-cbtrk\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274257 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274236    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1477b2-7e81-4df4-b942-ca04c61934aa-istiod-ca-cert\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274462 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274411    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-certs\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.274845 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274821    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-data\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.275004 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274968    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-socket\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.275004 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274915    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1477b2-7e81-4df4-b942-ca04c61934aa-istiod-ca-cert\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.275231 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.274825    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-credential-socket\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.276683 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.276655    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-envoy\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.276792 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.276766    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-podinfo\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.340113 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.340078    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-token\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.340113 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.340105    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbtrk\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-kube-api-access-cbtrk\") pod \"e2e-ait-90774e61-openshift-default-5f66579499-5f2mm\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") " pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.380514 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.380466    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:03.833644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.833614    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"]
Nov 03 19:02:03.861043 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:02:03.861005    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1477b2_7e81_4df4_b942_ca04c61934aa.slice/crio-4e164747d1eb66ce2582a6f9312a7e01cf1c717f866dee31cc11659696911103 WatchSource:0}: Error finding container 4e164747d1eb66ce2582a6f9312a7e01cf1c717f866dee31cc11659696911103: Status 404 returned error can't find the container with id 4e164747d1eb66ce2582a6f9312a7e01cf1c717f866dee31cc11659696911103
Nov 03 19:02:03.868969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.867807    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:03.868969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.867909    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:03.868969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.867949    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:03.971544 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:03.971520    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:02:04.089656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.089600    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd0df80-fd02-4b89-aa5f-11eb2559e390-kube-api-access-w2vw7\" (UniqueName: \"kubernetes.io/projected/ccd0df80-fd02-4b89-aa5f-11eb2559e390-kube-api-access-w2vw7\") pod \"ccd0df80-fd02-4b89-aa5f-11eb2559e390\" (UID: \"ccd0df80-fd02-4b89-aa5f-11eb2559e390\") "
Nov 03 19:02:04.093056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.093027    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd0df80-fd02-4b89-aa5f-11eb2559e390-kube-api-access-w2vw7" pod "ccd0df80-fd02-4b89-aa5f-11eb2559e390" (UID: "ccd0df80-fd02-4b89-aa5f-11eb2559e390"). InnerVolumeSpecName "kube-api-access-w2vw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:04.153137 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.153100    4489 generic.go:356] "Generic (PLEG): container finished" podID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" containerID="465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8" exitCode=137
Nov 03 19:02:04.153137 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.153150    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:02:04.153397 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.153201    4489 scope.go:122] "RemoveContainer" containerID="465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8"
Nov 03 19:02:04.155251 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.155223    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" event={"ID":"1a1477b2-7e81-4df4-b942-ca04c61934aa","Type":"ContainerStarted","Data":"4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328"}
Nov 03 19:02:04.155383 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.155280    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" event={"ID":"1a1477b2-7e81-4df4-b942-ca04c61934aa","Type":"ContainerStarted","Data":"4e164747d1eb66ce2582a6f9312a7e01cf1c717f866dee31cc11659696911103"}
Nov 03 19:02:04.163837 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.163792    4489 scope.go:122] "RemoveContainer" containerID="465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8"
Nov 03 19:02:04.164271 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:02:04.164237    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8\": container with ID starting with 465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8 not found: ID does not exist" containerID="465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8"
Nov 03 19:02:04.164369 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.164282    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8"} err="failed to get container status \"465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8\": rpc error: code = NotFound desc = could not find container \"465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8\": container with ID starting with 465d5edbf9ac4854f225ab9cb1edbd9c43be171098eff2d838bb7f70d4924ab8 not found: ID does not exist"
Nov 03 19:02:04.191531 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.191498    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2vw7\" (UniqueName: \"kubernetes.io/projected/ccd0df80-fd02-4b89-aa5f-11eb2559e390-kube-api-access-w2vw7\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:04.283483 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.283387    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" podStartSLOduration=1.283372681 podStartE2EDuration="1.283372681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:04.279275829 +0000 UTC m=+1453.427970401" watchObservedRunningTime="2026-11-03 19:02:04.283372681 +0000 UTC m=+1453.432067315"
Nov 03 19:02:04.332479 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:02:04.332432    4489 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-714cf1\" is forbidden: User \"system:node:ip-10-0-143-45.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-45.us-west-2.compute.internal' and this object" podUID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" pod="opendatahub/test-curl-7315-714cf1"
Nov 03 19:02:04.381531 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.381498    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:04.995437 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:04.995404    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd0df80-fd02-4b89-aa5f-11eb2559e390" path="/var/lib/kubelet/pods/ccd0df80-fd02-4b89-aa5f-11eb2559e390/volumes"
Nov 03 19:02:05.155010 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.154980    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"]
Nov 03 19:02:05.161780 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.161747    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203353    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203391    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203501    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203543    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203685 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203674    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203719 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203693    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203757 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203729    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfl45\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-kube-api-access-qfl45\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203872 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203850    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.203919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.203897    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.217877 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.217843    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"]
Nov 03 19:02:05.304735 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.304661    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.304735 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.304691    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.304735 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.304717    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305006 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.304846    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305068 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305013    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305068 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305043    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305206 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305070    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305206 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305103    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfl45\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-kube-api-access-qfl45\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305206 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305107    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305206 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305134    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305206 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305178    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305463 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305227    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305463 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305357    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.305835 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.305812    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.307240 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.307222    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.307587 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.307572    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.368579 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.368543    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.368843 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.368823    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfl45\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-kube-api-access-qfl45\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.386989 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.386962    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:05.476970 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.476945    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:05.898023 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:02:05.897987    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5373c6d5_a7f0_4dc7_9580_336354deb1cb.slice/crio-bb09dd5735ce077b732dee79fddea48fd022e71835513956558f635b68bdef49 WatchSource:0}: Error finding container bb09dd5735ce077b732dee79fddea48fd022e71835513956558f635b68bdef49: Status 404 returned error can't find the container with id bb09dd5735ce077b732dee79fddea48fd022e71835513956558f635b68bdef49
Nov 03 19:02:05.902844 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.902804    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:05.902969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.902895    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:05.902969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.902937    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:05.905245 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:05.905222    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"]
Nov 03 19:02:06.175531 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.175490    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" event={"ID":"5373c6d5-a7f0-4dc7-9580-336354deb1cb","Type":"ContainerStarted","Data":"830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8"}
Nov 03 19:02:06.175976 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.175553    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" event={"ID":"5373c6d5-a7f0-4dc7-9580-336354deb1cb","Type":"ContainerStarted","Data":"bb09dd5735ce077b732dee79fddea48fd022e71835513956558f635b68bdef49"}
Nov 03 19:02:06.175976 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.175748    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:06.176795 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.176774    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:02:06.303032 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.302977    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podStartSLOduration=1.302962361 podStartE2EDuration="1.302962361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:06.301655801 +0000 UTC m=+1455.450350369" watchObservedRunningTime="2026-11-03 19:02:06.302962361 +0000 UTC m=+1455.451656932"
Nov 03 19:02:06.477377 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.477279    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:06.478417 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.478389    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:06.478559 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:06.478449    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:02:07.493185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:07.491210    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:07.493185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:07.491284    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:02:08.478765 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:08.478315    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:08.478765 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:08.478382    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:02:09.477437 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:09.477401    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:09.477846 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:09.477468    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:02:10.477719 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:10.477676    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:10.478243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:10.477762    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:02:11.477229 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:11.477191    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:11.477431 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:11.477269    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:02:12.477363 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:12.477328    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:02:12.477874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:12.477387    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:02:13.515169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:13.515125    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:13.515703 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:13.515554    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:13.516625 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:13.516600    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:02:14.996115 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:14.996082    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"]
Nov 03 19:02:14.999988 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:14.999971    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.054988 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.054952    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"]
Nov 03 19:02:15.072011 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.071975    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-certs\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072226 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072024    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-token\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072226 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072048    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-credential-socket\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072226 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072069    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-data\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072277    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-envoy\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072324    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-socket\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072459 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072433    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbv4\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-kube-api-access-dnbv4\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072463    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istiod-ca-cert\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.072553 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.072500    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-podinfo\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.103259 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.103230    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-30b92815-openshift-default-dockercfg-6c49b\""
Nov 03 19:02:15.214958 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.214919    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-certs\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.215249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.215228    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-token\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.215374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.215361    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-credential-socket\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.215461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.215450    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-data\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.215589 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.215576    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-envoy\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.215689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.215656    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-certs\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.216771 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.215664    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-socket\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.217030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.217010    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbv4\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-kube-api-access-dnbv4\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.217420 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.217403    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istiod-ca-cert\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.218161 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.218127    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-podinfo\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.218417 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.215944    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-socket\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.218512 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.216120    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-data\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.218591 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.218064    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istiod-ca-cert\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.218684 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.216709    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-credential-socket\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.221444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.221402    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-podinfo\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.226840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.226390    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-envoy\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.279599 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.279520    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-token\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.279730 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.279712    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbv4\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-kube-api-access-dnbv4\") pod \"e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") " pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.313454 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.313424    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:15.743058 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:02:15.743033    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a355f4a_4098_4c2d_94eb_5b1d1577e3a4.slice/crio-1d04fe05f000dd8458eeb2d906363f7a37cb4d9d14bc984addd8f382c974b549 WatchSource:0}: Error finding container 1d04fe05f000dd8458eeb2d906363f7a37cb4d9d14bc984addd8f382c974b549: Status 404 returned error can't find the container with id 1d04fe05f000dd8458eeb2d906363f7a37cb4d9d14bc984addd8f382c974b549
Nov 03 19:02:15.745760 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.745726    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:15.745849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.745796    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:15.745849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.745825    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:15.747258 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:15.747235    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"]
Nov 03 19:02:16.274078 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:16.274041    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" event={"ID":"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4","Type":"ContainerStarted","Data":"238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452"}
Nov 03 19:02:16.274554 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:16.274097    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" event={"ID":"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4","Type":"ContainerStarted","Data":"1d04fe05f000dd8458eeb2d906363f7a37cb4d9d14bc984addd8f382c974b549"}
Nov 03 19:02:16.313781 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:16.313751    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:16.319898 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:16.319877    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:16.404009 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:16.403959    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podStartSLOduration=2.403945639 podStartE2EDuration="2.403945639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:16.402264087 +0000 UTC m=+1465.550958669" watchObservedRunningTime="2026-11-03 19:02:16.403945639 +0000 UTC m=+1465.552640270"
Nov 03 19:02:17.279640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:17.279615    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:17.280545 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:17.280526    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:02:17.538017 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:02:17.537888    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:02:24.422762 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.422718    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"]
Nov 03 19:02:24.440502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.437435    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.494971 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.494935    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"]
Nov 03 19:02:24.537684 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.537654    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:02:24.537862 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.537654    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:02:24.537862 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.537714    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:02:24.537862 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.537716    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4qf6p\""
Nov 03 19:02:24.543509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.543482    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.543509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.543511    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtm42\" (UniqueName: \"kubernetes.io/projected/f67ee830-9aa5-4ad7-9741-38b9645c6639-kube-api-access-dtm42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.543689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.543540    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.543689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.543663    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.543689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.543686    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.543905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.543886    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.544008 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.543994    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ee830-9aa5-4ad7-9741-38b9645c6639-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.646395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.646363    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.646606 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.646450    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ee830-9aa5-4ad7-9741-38b9645c6639-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.646606 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.646497    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.646606 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.646520    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtm42\" (UniqueName: \"kubernetes.io/projected/f67ee830-9aa5-4ad7-9741-38b9645c6639-kube-api-access-dtm42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.646606 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.646557    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.646606 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.646601    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.646880 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.646748    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.647916 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.647884    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.649016 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.648991    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.649104 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.649037    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ee830-9aa5-4ad7-9741-38b9645c6639-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.649104 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.649092    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.649204 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.649160    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.713427 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.713359    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtm42\" (UniqueName: \"kubernetes.io/projected/f67ee830-9aa5-4ad7-9741-38b9645c6639-kube-api-access-dtm42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.714617 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.714590    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67ee830-9aa5-4ad7-9741-38b9645c6639-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-79xhw\" (UID: \"f67ee830-9aa5-4ad7-9741-38b9645c6639\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:24.792456 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:24.792428    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:25.218395 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:02:25.218366    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67ee830_9aa5_4ad7_9741_38b9645c6639.slice/crio-8a2907855a6a2d81e26d6aa2eb79dd0e516fe16f94812a8fb218dbce79ce6d61 WatchSource:0}: Error finding container 8a2907855a6a2d81e26d6aa2eb79dd0e516fe16f94812a8fb218dbce79ce6d61: Status 404 returned error can't find the container with id 8a2907855a6a2d81e26d6aa2eb79dd0e516fe16f94812a8fb218dbce79ce6d61
Nov 03 19:02:25.220578 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:25.220550    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"]
Nov 03 19:02:25.220813 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:25.220781    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:25.220886 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:25.220860    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:25.333268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:25.333230    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw" event={"ID":"f67ee830-9aa5-4ad7-9741-38b9645c6639","Type":"ContainerStarted","Data":"3388e0f7e645d96f47ab2ec998328b1144960913bf0ee0b9d309d38275e47505"}
Nov 03 19:02:25.333450 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:25.333292    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw" event={"ID":"f67ee830-9aa5-4ad7-9741-38b9645c6639","Type":"ContainerStarted","Data":"8a2907855a6a2d81e26d6aa2eb79dd0e516fe16f94812a8fb218dbce79ce6d61"}
Nov 03 19:02:25.333513 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:25.333459    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:25.460710 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:25.460653    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw" podStartSLOduration=1.460639611 podStartE2EDuration="1.460639611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:25.458119635 +0000 UTC m=+1474.606814207" watchObservedRunningTime="2026-11-03 19:02:25.460639611 +0000 UTC m=+1474.609334182"
Nov 03 19:02:26.343030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:26.342979    4489 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-79xhw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:02:26.343325 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:26.343039    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw" podUID="f67ee830-9aa5-4ad7-9741-38b9645c6639" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:02:29.341880 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:29.341844    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-79xhw"
Nov 03 19:02:34.097443 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.097366    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"]
Nov 03 19:02:34.102775 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.102744    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.173998    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79xc\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-kube-api-access-m79xc\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174236 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174114    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174236 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174218    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174388 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174367    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c55759-2371-4366-9749-b11edeb3a7dc-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174440 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174424    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174521 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174507    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174601    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174663 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174641    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.174749 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.174735    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.212686 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.212652    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"]
Nov 03 19:02:34.276593 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276563    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c55759-2371-4366-9749-b11edeb3a7dc-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.276773 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276598    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.276773 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276667    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.276773 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276695    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.276773 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276709    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.276955 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276843    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.276955 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276925    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79xc\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-kube-api-access-m79xc\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.277040 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.276981    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.277040 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.277013    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.277909 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.277533    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c55759-2371-4366-9749-b11edeb3a7dc-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.277909 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.277575    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.277909 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.277633    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.277909 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.277648    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.277909 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.277852    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.279635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.279610    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.279745 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.279727    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.283903 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.283884    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"]
Nov 03 19:02:34.287231 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.287215    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.342509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.341253    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.349942 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.349663    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"]
Nov 03 19:02:34.350093 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.350030    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79xc\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-kube-api-access-m79xc\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.378705 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.378663    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.378871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.378726    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.378871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.378825    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.379013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.378993    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.379087 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.379059    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.379252 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.379227    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.379408 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.379382    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.379484 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.379467    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfv5\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-kube-api-access-6cfv5\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.379532 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.379507    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.417131 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.417093    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:34.481762 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.481729    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482330 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482307    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482400 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482351    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482494 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482478    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482552    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482623 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482602    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482695 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482680    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482775 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482758    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfv5\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-kube-api-access-6cfv5\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482827 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482801    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.482983 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.482240    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.483472 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.483448    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.483577 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.483552    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.483680 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.483657    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.484241 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.484219    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.486121 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.486080    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.487717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.487697    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.549256 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.549222    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.555519 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.555489    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfv5\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-kube-api-access-6cfv5\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.612025 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.611957    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:34.853329 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:02:34.853293    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c55759_2371_4366_9749_b11edeb3a7dc.slice/crio-c61fcbacb98b815da3a6715b7318131735ff88c79ea3d7ad14069c7dd3200836 WatchSource:0}: Error finding container c61fcbacb98b815da3a6715b7318131735ff88c79ea3d7ad14069c7dd3200836: Status 404 returned error can't find the container with id c61fcbacb98b815da3a6715b7318131735ff88c79ea3d7ad14069c7dd3200836
Nov 03 19:02:34.856321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.856284    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:34.856439 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.856367    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:34.856439 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.856412    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:34.857639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.857618    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"]
Nov 03 19:02:34.993427 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.993391    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:34.993564 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.993467    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:34.993564 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:34.993509    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:35.001232 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.001208    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"]
Nov 03 19:02:35.088179 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.086889    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"]
Nov 03 19:02:35.092918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.092891    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.147995 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.147913    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"]
Nov 03 19:02:35.195179 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195114    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrm4\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-kube-api-access-dmrm4\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195210    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195339    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195372    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195501    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195540    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195564    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195612    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.195863 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.195640    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297369    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297423    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297547    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrm4\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-kube-api-access-dmrm4\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297575    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297753 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297687    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297753 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297723    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297906 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297889    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.297969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297939    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.298026 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.297964    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.298026 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.298001    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.298325 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.298299    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.298560 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.298540    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.298675 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.298620    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.298765 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.298744    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.300686 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.300661    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.301432 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.301406    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.368747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.368716    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrm4\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-kube-api-access-dmrm4\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.368747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.368731    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.407062 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.407022    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" event={"ID":"4e29bf55-1ada-4be6-a5ff-af33f146b8a4","Type":"ContainerStarted","Data":"6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46"}
Nov 03 19:02:35.407257 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.407093    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" event={"ID":"4e29bf55-1ada-4be6-a5ff-af33f146b8a4","Type":"ContainerStarted","Data":"9f98df4b8dca2f58ad531b8c566a7d80d5c1010bc83a5e6ecc3f8230d5cd8c30"}
Nov 03 19:02:35.410311 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.410284    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" event={"ID":"a7c55759-2371-4366-9749-b11edeb3a7dc","Type":"ContainerStarted","Data":"a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814"}
Nov 03 19:02:35.410447 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.410325    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" event={"ID":"a7c55759-2371-4366-9749-b11edeb3a7dc","Type":"ContainerStarted","Data":"c61fcbacb98b815da3a6715b7318131735ff88c79ea3d7ad14069c7dd3200836"}
Nov 03 19:02:35.411622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.411606    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:35.421048 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.421026    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:35.423032 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.422998    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:35.423032 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.423043    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:35.534669 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.534619    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podStartSLOduration=1.534604666 podStartE2EDuration="1.534604666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:35.531676428 +0000 UTC m=+1484.680371022" watchObservedRunningTime="2026-11-03 19:02:35.534604666 +0000 UTC m=+1484.683299237"
Nov 03 19:02:35.612644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.612607    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:35.645507 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.645478    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"]
Nov 03 19:02:35.648830 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.648810    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.659248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.659127    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podStartSLOduration=1.659114178 podStartE2EDuration="1.659114178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:35.656321809 +0000 UTC m=+1484.805016400" watchObservedRunningTime="2026-11-03 19:02:35.659114178 +0000 UTC m=+1484.807808756"
Nov 03 19:02:35.712924 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.712884    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"]
Nov 03 19:02:35.806476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.806438    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd596047-c356-4d71-b722-bc90cdbded25-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.806668 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.806521    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.806668 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.806555    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.806668 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.806589    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4q9\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-kube-api-access-rw4q9\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.806784 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.806763    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.806829 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.806785    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.806928 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.806914    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd596047-c356-4d71-b722-bc90cdbded25-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.807157 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.807107    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.807293 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.807256    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.849038 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.844585    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"]
Nov 03 19:02:35.892590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.892546    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:35.892758 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.892650    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:35.892758 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.892711    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:35.920153 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920050    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920153 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920114    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920253    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd596047-c356-4d71-b722-bc90cdbded25-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920349    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920412    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920467    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd596047-c356-4d71-b722-bc90cdbded25-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920533    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920565    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.920849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920607    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4q9\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-kube-api-access-rw4q9\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.921008 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.920991    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.922607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.921600    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd596047-c356-4d71-b722-bc90cdbded25-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.922607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.921718    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.922607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.921959    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.922607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.922277    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.925975 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.925711    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:35.926173 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:35.926096    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd596047-c356-4d71-b722-bc90cdbded25-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:36.007526 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.007494    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4q9\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-kube-api-access-rw4q9\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:36.012318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.012299    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:36.273574 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.273450    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:36.426722 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.426685    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:36.428419 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.428378    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:36.438370 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.437896    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" event={"ID":"5ded5c82-3bca-4fa0-b294-1864c3dff0bd","Type":"ContainerStarted","Data":"6f708128a4ccf7863c5cf71a3447f45546d4347695810c4e335c3711ecd61227"}
Nov 03 19:02:36.438370 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.437990    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" event={"ID":"5ded5c82-3bca-4fa0-b294-1864c3dff0bd","Type":"ContainerStarted","Data":"27df0d8f3f1d4eb6a02f082e8fce6f5a8cf86db50b7364c965b2c5ba3183564b"}
Nov 03 19:02:36.619482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.619394    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:36.619482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.619454    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:36.815336 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.815269    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podStartSLOduration=1.815244731 podStartE2EDuration="1.815244731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:36.616395175 +0000 UTC m=+1485.765089745" watchObservedRunningTime="2026-11-03 19:02:36.815244731 +0000 UTC m=+1485.963939302"
Nov 03 19:02:36.817843 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.817818    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"]
Nov 03 19:02:36.819188 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:02:36.819127    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd596047_c356_4d71_b722_bc90cdbded25.slice/crio-08af116bf05ebbcd73f79a5a40efab8bfd881f32535343e07385d21a747a554e WatchSource:0}: Error finding container 08af116bf05ebbcd73f79a5a40efab8bfd881f32535343e07385d21a747a554e: Status 404 returned error can't find the container with id 08af116bf05ebbcd73f79a5a40efab8bfd881f32535343e07385d21a747a554e
Nov 03 19:02:36.822518 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.822480    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:36.822596 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.822569    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:36.822632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:36.822608    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:02:37.433227 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.429880    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:37.433227 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.430001    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:37.433227 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.430040    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:37.433227 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.431336    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:37.433227 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.431375    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:37.521638 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.469390    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" event={"ID":"dd596047-c356-4d71-b722-bc90cdbded25","Type":"ContainerStarted","Data":"58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3"}
Nov 03 19:02:37.521638 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.469471    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" event={"ID":"dd596047-c356-4d71-b722-bc90cdbded25","Type":"ContainerStarted","Data":"08af116bf05ebbcd73f79a5a40efab8bfd881f32535343e07385d21a747a554e"}
Nov 03 19:02:37.807394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.782451    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:37.811789 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:37.811566    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:38.296168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.295328    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:38.296168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.295483    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:38.296168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.295529    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:38.413173 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.412055    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:38.413173 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.412120    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:38.422352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.422322    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:38.422630 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.422605    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:38.613473 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.613375    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:38.614087 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.614030    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:38.817169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.813239    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:02:38.830225 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.826230    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:38.857182 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.853221    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:38.915170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.913233    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:38.915170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.913620    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:38.915170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.913912    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:02:38.935165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:38.929295    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.279905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.279806    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:39.280184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.280122    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:39.413559 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.413523    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:39.413832 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.413803    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:39.426169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.425821    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:39.426169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.425901    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:39.620230 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.616551    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:39.620230 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.616610    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:39.653637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.643264    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.704196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.676113    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.772894 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.759236    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.836220 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.836183    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:02:39.836979 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.836959    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 19:02:39.838186 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.838136    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:02:39.842348 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.842312    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:02:39.873252 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.868363    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.896199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.885551    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.897250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.897226    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 19:02:39.916185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.912064    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.936327 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.936282    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:39.937747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.937722    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:02:39.960170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.941074    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:40.332506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.941122    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:02:40.332506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.332456    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:40.332765 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.332516    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:40.332765 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.028749    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podStartSLOduration=5.028724437 podStartE2EDuration="5.028724437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:37.584737948 +0000 UTC m=+1486.733432532" watchObservedRunningTime="2026-11-03 19:02:40.028724437 +0000 UTC m=+1489.177419009"
Nov 03 19:02:40.347630 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.340329    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:02:40.347630 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.340423    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:02:40.347630 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:39.941161    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:02:40.347897 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.347875    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:02:40.444334 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.444302    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:40.444602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.444571    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:40.446119 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.446095    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:40.446314 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.446291    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:40.679171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.662279    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:40.708107 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.708064    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:40.708433 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.708383    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:40.713163 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.712013    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:02:40.798169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.777389    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:40.839629 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.799442    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:40.839941 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.839904    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:02:40.840221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.840196    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:40.864633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.860116    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:02:40.891170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.876266    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:02:40.891509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.891471    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 19:02:40.892964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.892941    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:02:40.920451 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.920388    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:02:40.960665 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.960569    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Nov 03 19:02:40.962680 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.962655    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 19:02:40.986178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:40.965263    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.269217 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.268396    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:02:41.397010 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.396976    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:02:41.405247 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.405202    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:41.462992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.409229    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:41.476248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.476193    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.497275 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.480746    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:41.537460 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.537373    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:41.548474 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.537276    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.565590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.565544    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:41.569279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.569232    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:41.624556 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.624455    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:41.624830 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.624800    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:41.685384 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.685343    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.747382 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.747345    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:41.747659 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.747617    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.765372 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.765250    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:41.765372 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.765308    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.815222 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.805879    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:02:41.909274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.909223    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.920350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.910749    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:41.920350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.911472    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:02:41.948171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.920662    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:02:41.984169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.976628    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:42.008169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.984788    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:42.064085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:41.986209    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:42.288169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.280697    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:42.351464 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.288222    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:02:42.351974 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.292294    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:42.352187 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.352136    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:42.398578 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.398536    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:42.420602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.420568    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:42.420896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.420850    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:42.555228 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.487444    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:42.555228 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.487505    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:42.555228 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.524498    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 19:02:42.616506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.616467    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:42.616778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.616750    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:42.851030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.850927    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:42.855514 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.853070    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:02:42.961196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:42.952237    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:43.062262 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.062220    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:43.075171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.071725    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:02:43.116816 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.076615    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:02:43.117348 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.076654    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 19:02:43.117789 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.076675    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:02:43.118221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.090211    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Nov 03 19:02:43.118622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.093943    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:43.315303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.315251    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 19:02:43.357605 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.357362    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 19:02:43.371249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.364407    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:43.371249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.364460    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:43.444170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.444007    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:43.446414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.445312    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:43.446414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.445360    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:43.446817 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.446704    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:43.446817 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.446762    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:43.547895 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.547855    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:43.613572 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.613528    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:43.613874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.613824    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:43.876168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.856257    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:43.969834 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.969791    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:43.971083 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:43.971054    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:02:44.255169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.248307    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:02:44.255169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.224065    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.340661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.340613    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:02:44.344752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.341653    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:02:44.344752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.343467    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 03 19:02:44.344752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.343733    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:02:44.347292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.345272    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:44.347292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.345312    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:44.387169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.360060    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.432169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.420518    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:44.432169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.420574    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:44.433884 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.433844    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:44.434123 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.434088    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:44.522025 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.521917    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:02:44.553567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.553525    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.617167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.614494    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:44.617167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.614551    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:44.861477 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.861375    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.985715 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.985670    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.986658 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.986234    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:44.987628 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:44.987605    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:02:45.345404 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.288954    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:45.345582 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.289022    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:02:45.347580 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.347093    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:45.347580 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.347175    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:45.379183 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.372452    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:45.379686 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.344987    4489 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:45.387294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.374517    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:45.387642 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.387608    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:45.419174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.413716    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:02:45.419174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.413793    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:02:45.419174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.413937    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:02:45.419174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.413987    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:02:45.419174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.414033    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:02:45.419174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.414081    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:02:45.423530 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.423497    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:45.423762 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.423732    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:45.428689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.428462    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:45.428908 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.428882    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:02:45.636218 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.635544    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:02:45.636218 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.635609    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:02:45.743615 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.743585    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:02:45.814645 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.814617    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 19:02:45.814915 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.814901    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:02:45.815053 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.815041    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:02:45.815224 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.815209    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 19:02:45.815368 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:45.815356    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 19:02:46.299364 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:46.299321    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:46.299986 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:46.299937    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:46.418174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:46.413324    4489 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:02:46.418174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:46.413382    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 19:02:47.275116 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.275081    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:02:47.275418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.275385    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 19:02:47.419995 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.419960    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:47.420593 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.420560    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:47.561045 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:02:47.560926    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:02:47.665165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.657325    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:47.665165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.657431    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:47.923233 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.912563    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:47.923233 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.914037    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:47.936825 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:47.936793    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:02:48.478597 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.478247    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:48.478597 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.478304    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:48.539197 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.535541    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:48.539197 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.537562    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:48.547298 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.547253    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:02:48.641289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.641243    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:48.641780 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.641740    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:48.642403 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:48.642383    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:02:49.348338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:49.348303    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:02:49.348612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:49.348582    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:02:49.941182 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:49.941132    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:49.941661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:49.941548    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:02:49.942397 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:02:49.942375    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:03:17.669702 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:03:17.669663    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:03:19.051192 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:19.042243    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:19.555170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:19.554513    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:19.741179 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:19.740243    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:19.812226 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:19.811761    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:19.924173 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:19.923219    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:19.971170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:19.970240    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.012170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.011304    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:03:20.026169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.020758    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.044178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.043235    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.044178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.043697    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 19:03:20.154250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.136234    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.366183 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.364447    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.431181 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.416676    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.576079 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.576020    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.577278 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.577243    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:03:20.759924 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.754982    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.780705 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.780674    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:03:20.910169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.814814    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.910169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.903092    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:03:20.910169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.905859    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:03:20.943951 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.923922    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:20.943951 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:20.924238    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:03:21.087502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.038278    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:21.087502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.044903    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:03:21.087502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.050659    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v" podUID="e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Nov 03 19:03:21.223004 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.146655    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:03:21.435865 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.435828    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467294    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467366    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467418    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467448    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467477    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467507    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467537    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:03:21.469168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.467572    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:03:21.495778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.495752    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 19:03:21.505844 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.501578    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:03:21.516243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.516211    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:03:21.517706 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.517680    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:03:21.522620 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:21.518999    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:03:28.439871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:28.439841    4489 generic.go:356] "Generic (PLEG): container finished" podID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerID="4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328" exitCode=0
Nov 03 19:03:28.440437 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:28.439896    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" event={"ID":"1a1477b2-7e81-4df4-b942-ca04c61934aa","Type":"ContainerDied","Data":"4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328"}
Nov 03 19:03:28.440437 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:28.440413    4489 scope.go:122] "RemoveContainer" containerID="4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328"
Nov 03 19:03:28.442052 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:28.442026    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:28.442137 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:28.442096    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:28.442137 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:28.442126    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:29.473656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:29.473608    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" event={"ID":"1a1477b2-7e81-4df4-b942-ca04c61934aa","Type":"ContainerStarted","Data":"9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b"}
Nov 03 19:03:30.382510 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:30.382475    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:03:30.382733 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:30.382698    4489 patch_prober.go:29] interesting pod/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:03:30.382803 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:30.382748    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:03:31.381182 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:31.381127    4489 patch_prober.go:29] interesting pod/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:03:31.381628 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:31.381223    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:03:32.381771 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:32.381733    4489 patch_prober.go:29] interesting pod/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:03:32.382212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:32.381805    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:03:33.381675 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:33.381634    4489 patch_prober.go:29] interesting pod/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:03:33.381859 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:33.381707    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:03:34.381248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:34.381210    4489 patch_prober.go:29] interesting pod/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:03:34.381430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:34.381266    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:03:35.385911 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:35.385881    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:03:35.386427 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:35.386250    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:03:35.387092 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:35.387071    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:03:46.716837 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.716797    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"]
Nov 03 19:03:46.719915 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.719897    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.774609 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.774576    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h6x4t\""
Nov 03 19:03:46.790814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.790775    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791021 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.790992    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791087 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.791054    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791201 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.791176    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791306 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.791289    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4b95\" (UniqueName: \"kubernetes.io/projected/f837beb4-df93-41a8-ad75-0381fc6c0995-kube-api-access-b4b95\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791372 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.791356    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791573 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.791543    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791722 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.791605    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f837beb4-df93-41a8-ad75-0381fc6c0995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.791722 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.791701    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.840953 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.840916    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"]
Nov 03 19:03:46.893844 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.893800    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4b95\" (UniqueName: \"kubernetes.io/projected/f837beb4-df93-41a8-ad75-0381fc6c0995-kube-api-access-b4b95\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894061 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.893903    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894061 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.893995    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894061 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894043    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f837beb4-df93-41a8-ad75-0381fc6c0995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894089    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894136    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894376 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894285    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894376 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894328    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894376 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894364    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894790    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894933 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894829    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.894933 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.894897    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f837beb4-df93-41a8-ad75-0381fc6c0995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.895132 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.895104    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.895541 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.895511    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.897032 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.896980    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.897862 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.897835    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.957918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.957884    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f837beb4-df93-41a8-ad75-0381fc6c0995-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:46.958062 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:46.958015    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4b95\" (UniqueName: \"kubernetes.io/projected/f837beb4-df93-41a8-ad75-0381fc6c0995-kube-api-access-b4b95\") pod \"maas-default-gateway-openshift-default-7854949c9-zlt64\" (UID: \"f837beb4-df93-41a8-ad75-0381fc6c0995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:47.032950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.032869    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:47.464555 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:03:47.464511    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf837beb4_df93_41a8_ad75_0381fc6c0995.slice/crio-0b56855aac51431df0df4e6a02eec51cb0b80d334975264ef2cc59d4ef9ebfde WatchSource:0}: Error finding container 0b56855aac51431df0df4e6a02eec51cb0b80d334975264ef2cc59d4ef9ebfde: Status 404 returned error can't find the container with id 0b56855aac51431df0df4e6a02eec51cb0b80d334975264ef2cc59d4ef9ebfde
Nov 03 19:03:47.467393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.467362    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:47.467493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.467426    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:47.467493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.467460    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:47.469632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.469610    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"]
Nov 03 19:03:47.536691 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:03:47.536654    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:03:47.601694 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.601661    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" event={"ID":"f837beb4-df93-41a8-ad75-0381fc6c0995","Type":"ContainerStarted","Data":"6465d3a44cff65774e61677286ff6d674b71c1cb7cb6aa1c628f87242fef85bc"}
Nov 03 19:03:47.601900 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.601719    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" event={"ID":"f837beb4-df93-41a8-ad75-0381fc6c0995","Type":"ContainerStarted","Data":"0b56855aac51431df0df4e6a02eec51cb0b80d334975264ef2cc59d4ef9ebfde"}
Nov 03 19:03:47.731869 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:47.731760    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podStartSLOduration=1.7317386830000001 podStartE2EDuration="1.731738683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:47.728897975 +0000 UTC m=+1556.877592558" watchObservedRunningTime="2026-11-03 19:03:47.731738683 +0000 UTC m=+1556.880433255"
Nov 03 19:03:48.034086 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:48.034006    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:48.035657 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:48.035635    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:48.035744 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:48.035691    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:49.034075 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.034039    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:49.034580 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.034097    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:49.720623 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.720584    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"]
Nov 03 19:03:49.723703 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.723684    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839235    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839277    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fm57\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-kube-api-access-2fm57\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839329    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10f023-91be-4a7b-b2c8-2ff070481e84-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839368    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839413    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839480    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839502    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839588    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.840716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.839613    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.934249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.934218    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"]
Nov 03 19:03:49.940928 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.940887    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941076 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.940930    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941076 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.940980    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941076 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.940995    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fm57\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-kube-api-access-2fm57\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941076 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.941049    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10f023-91be-4a7b-b2c8-2ff070481e84-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.941086    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.941195    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.941259    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.941282    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.941881 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.941849    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.942054 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.942038    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.943382 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.943158    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.943673 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.943582    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10f023-91be-4a7b-b2c8-2ff070481e84-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.943891 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.943867    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.944864 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.944844    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:49.945388 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:49.945365    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:50.006072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.005996    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fm57\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-kube-api-access-2fm57\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:50.008102 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.008067    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:50.033715 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.033685    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:50.033845 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.033759    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:50.037051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.037032    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:50.512210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.508771    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"]
Nov 03 19:03:50.577242 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:03:50.571359    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a10f023_91be_4a7b_b2c8_2ff070481e84.slice/crio-00bc58f0350c76b230b217062eee9f86cbb18815d906fe9dbea43b87bbfdf827 WatchSource:0}: Error finding container 00bc58f0350c76b230b217062eee9f86cbb18815d906fe9dbea43b87bbfdf827: Status 404 returned error can't find the container with id 00bc58f0350c76b230b217062eee9f86cbb18815d906fe9dbea43b87bbfdf827
Nov 03 19:03:50.589315 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.589273    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:50.589576 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.589551    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:50.589719 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.589703    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:03:50.687169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:50.685281    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" event={"ID":"5a10f023-91be-4a7b-b2c8-2ff070481e84","Type":"ContainerStarted","Data":"00bc58f0350c76b230b217062eee9f86cbb18815d906fe9dbea43b87bbfdf827"}
Nov 03 19:03:51.033620 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:51.033585    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:51.033896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:51.033869    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:51.715829 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:51.715287    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" event={"ID":"5a10f023-91be-4a7b-b2c8-2ff070481e84","Type":"ContainerStarted","Data":"de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579"}
Nov 03 19:03:51.903951 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:51.903893    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podStartSLOduration=2.903878229 podStartE2EDuration="2.903878229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:51.899816329 +0000 UTC m=+1561.048510902" watchObservedRunningTime="2026-11-03 19:03:51.903878229 +0000 UTC m=+1561.052572799"
Nov 03 19:03:52.033694 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:52.033610    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:52.033694 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:52.033663    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:52.037839 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:52.037817    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:52.039214 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:52.039195    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:52.039339 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:52.039320    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:03:53.033891 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:53.033856    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:53.034375 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:53.034351    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:53.037360 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:53.037338    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:53.037515 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:53.037499    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:03:54.034052 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:54.034011    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:54.034490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:54.034082    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:54.038393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:54.038363    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:54.038515 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:54.038424    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:03:55.033683 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:55.033652    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:03:55.033922 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:55.033702    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:03:55.037996 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:55.037961    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:55.038452 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:55.038001    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:03:56.038205 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:56.037746    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:56.038205 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:56.037802    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:03:57.037840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:57.037804    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:57.038123 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:57.038094    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:03:57.039830 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:57.039798    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:03:57.040158 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:57.039847    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:03:57.134864 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:57.134833    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:57.136205 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:57.136126    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:57.137844 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:57.137820    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64"
Nov 03 19:03:58.037421 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:58.037383    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:58.037615 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:58.037452    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:03:59.042313 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:59.042282    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:59.042806 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:59.042649    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:03:59.043481 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:03:59.043461    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:04:08.871958 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:08.871869    4489 generic.go:356] "Generic (PLEG): container finished" podID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerID="238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452" exitCode=0
Nov 03 19:04:08.871958 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:08.871945    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" event={"ID":"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4","Type":"ContainerDied","Data":"238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452"}
Nov 03 19:04:08.872458 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:08.872411    4489 scope.go:122] "RemoveContainer" containerID="238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452"
Nov 03 19:04:08.874063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:08.874014    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:04:08.874120 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:08.874105    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:04:08.874190 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:08.874168    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:04:09.882174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:09.882109    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" event={"ID":"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4","Type":"ContainerStarted","Data":"3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1"}
Nov 03 19:04:10.314355 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:10.314312    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:04:10.314634 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:10.314605    4489 patch_prober.go:29] interesting pod/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:04:10.314694 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:10.314672    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:04:11.314449 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:11.314401    4489 patch_prober.go:29] interesting pod/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:04:11.314850 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:11.314470    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:04:12.314409 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:12.314370    4489 patch_prober.go:29] interesting pod/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:04:12.314601 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:12.314436    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:04:13.314042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:13.314009    4489 patch_prober.go:29] interesting pod/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:04:13.314252 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:13.314076    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:04:14.314060 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:14.314022    4489 patch_prober.go:29] interesting pod/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:04:14.314518 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:14.314089    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:04:15.497393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:15.497358    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:04:15.497893 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:15.497872    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:04:15.499459 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:15.499437    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:04:17.533729 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:04:17.533697    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:04:43.799861 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:43.799820    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"]
Nov 03 19:04:43.800388 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:43.800122    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy" containerID="cri-o://9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b" gracePeriod=30
Nov 03 19:04:47.532995 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:04:47.532956    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:04:49.101351 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.101328    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:04:49.179349 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179318    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-socket\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.179524 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179420    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-certs\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.179524 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179461    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-data\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.179661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179543    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-podinfo\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.179661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179620    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-kube-api-access-cbtrk\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-kube-api-access-cbtrk\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.179768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179673    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-credential-socket\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.179768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179670    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-socket" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:04:49.179768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179708    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-token\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-token\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.180212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179943    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-data" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:04:49.180212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.179955    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-credential-socket" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:04:49.180212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.180058    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-certs" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:04:49.180411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.180023    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1477b2-7e81-4df4-b942-ca04c61934aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1477b2-7e81-4df4-b942-ca04c61934aa-istiod-ca-cert\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.180494 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.180473    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-envoy\") pod \"1a1477b2-7e81-4df4-b942-ca04c61934aa\" (UID: \"1a1477b2-7e81-4df4-b942-ca04c61934aa\") "
Nov 03 19:04:49.180590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.180566    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1477b2-7e81-4df4-b942-ca04c61934aa-istiod-ca-cert" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:04:49.181356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.181154    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.181356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.181209    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1477b2-7e81-4df4-b942-ca04c61934aa-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.181356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.181262    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.181356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.181296    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.181356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.181344    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.183057 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.183032    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-kube-api-access-cbtrk" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "kube-api-access-cbtrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:04:49.183458 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.183428    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-envoy" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:04:49.183458 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.183439    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-podinfo" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:04:49.183626 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.183505    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-token" pod "1a1477b2-7e81-4df4-b942-ca04c61934aa" (UID: "1a1477b2-7e81-4df4-b942-ca04c61934aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:04:49.221800 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.221766    4489 generic.go:356] "Generic (PLEG): container finished" podID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerID="9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b" exitCode=0
Nov 03 19:04:49.221919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.221812    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" event={"ID":"1a1477b2-7e81-4df4-b942-ca04c61934aa","Type":"ContainerDied","Data":"9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b"}
Nov 03 19:04:49.221919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.221835    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"
Nov 03 19:04:49.221919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.221884    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm" event={"ID":"1a1477b2-7e81-4df4-b942-ca04c61934aa","Type":"ContainerDied","Data":"4e164747d1eb66ce2582a6f9312a7e01cf1c717f866dee31cc11659696911103"}
Nov 03 19:04:49.221919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.221914    4489 scope.go:122] "RemoveContainer" containerID="9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b"
Nov 03 19:04:49.233997 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.233961    4489 scope.go:122] "RemoveContainer" containerID="4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328"
Nov 03 19:04:49.242882 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.242861    4489 scope.go:122] "RemoveContainer" containerID="9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b"
Nov 03 19:04:49.243194 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:04:49.243174    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b\": container with ID starting with 9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b not found: ID does not exist" containerID="9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b"
Nov 03 19:04:49.243262 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.243202    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b"} err="failed to get container status \"9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b\": rpc error: code = NotFound desc = could not find container \"9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b\": container with ID starting with 9322ad216975a241213bd6db0c62bde3fa675b835bddf5e45ecff096188a1d3b not found: ID does not exist"
Nov 03 19:04:49.243262 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.243223    4489 scope.go:122] "RemoveContainer" containerID="4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328"
Nov 03 19:04:49.243498 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:04:49.243484    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328\": container with ID starting with 4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328 not found: ID does not exist" containerID="4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328"
Nov 03 19:04:49.243539 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.243502    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328"} err="failed to get container status \"4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328\": rpc error: code = NotFound desc = could not find container \"4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328\": container with ID starting with 4c333b3264a7cd6714536443c8ef5c4d05df82b8d3be4d065de8d6ef6a774328 not found: ID does not exist"
Nov 03 19:04:49.282250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.282227    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.282250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.282249    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.282386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.282260    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbtrk\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-kube-api-access-cbtrk\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.282386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.282271    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1477b2-7e81-4df4-b942-ca04c61934aa-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:49.396991 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.396914    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"]
Nov 03 19:04:49.400155 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:49.400121    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-90774e61-openshift-default-5f66579499-5f2mm"]
Nov 03 19:04:50.996525 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:50.996494    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" path="/var/lib/kubelet/pods/1a1477b2-7e81-4df4-b942-ca04c61934aa/volumes"
Nov 03 19:04:57.662815 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:57.662784    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"]
Nov 03 19:04:57.663229 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:04:57.663004    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy" containerID="cri-o://3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1" gracePeriod=30
Nov 03 19:05:00.500012 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:00.499966    4489 patch_prober.go:29] interesting pod/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:05:00.500430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:00.500037    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:05:02.973375 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:02.973349    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:05:03.109992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.109957    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-data\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110075    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-envoy\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110135    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istiod-ca-cert\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110283 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110225    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-socket\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110331 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110271    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-data" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:05:03.110331 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110287    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-podinfo\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110368    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-kube-api-access-dnbv4\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-kube-api-access-dnbv4\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110467 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110410    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-token\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-token\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110525 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110476    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-socket" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:05:03.110525 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110499    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-credential-socket\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110636 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110563    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-certs\") pod \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\" (UID: \"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4\") "
Nov 03 19:05:03.110636 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110597    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istiod-ca-cert" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:05:03.111000 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110970    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-certs" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:05:03.111000 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.110973    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-credential-socket" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:05:03.111404 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.111378    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.111506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.111406    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.111506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.111422    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.111506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.111436    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.111506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.111448    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.112595 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.112557    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-envoy" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:05:03.113070 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.113050    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-podinfo" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:05:03.113280 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.113259    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-kube-api-access-dnbv4" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "kube-api-access-dnbv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:05:03.113335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.113281    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-token" pod "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" (UID: "6a355f4a-4098-4c2d-94eb-5b1d1577e3a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:05:03.212767 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.212677    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.212767 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.212715    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.212767 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.212726    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnbv4\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-kube-api-access-dnbv4\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.212767 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.212735    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:05:03.349901 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.349867    4489 generic.go:356] "Generic (PLEG): container finished" podID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerID="3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1" exitCode=0
Nov 03 19:05:03.350068 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.349916    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"
Nov 03 19:05:03.350068 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.349953    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" event={"ID":"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4","Type":"ContainerDied","Data":"3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1"}
Nov 03 19:05:03.350068 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.350009    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6" event={"ID":"6a355f4a-4098-4c2d-94eb-5b1d1577e3a4","Type":"ContainerDied","Data":"1d04fe05f000dd8458eeb2d906363f7a37cb4d9d14bc984addd8f382c974b549"}
Nov 03 19:05:03.350068 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.350032    4489 scope.go:122] "RemoveContainer" containerID="3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1"
Nov 03 19:05:03.360657 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.360635    4489 scope.go:122] "RemoveContainer" containerID="238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452"
Nov 03 19:05:03.372196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.372166    4489 scope.go:122] "RemoveContainer" containerID="3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1"
Nov 03 19:05:03.372473 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:03.372453    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1\": container with ID starting with 3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1 not found: ID does not exist" containerID="3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1"
Nov 03 19:05:03.372523 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.372482    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1"} err="failed to get container status \"3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1\": rpc error: code = NotFound desc = could not find container \"3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1\": container with ID starting with 3dc4ccead03eeb67883ed3932a059bf157d89d2b07bcbf11f6207cf929eabcc1 not found: ID does not exist"
Nov 03 19:05:03.372523 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.372499    4489 scope.go:122] "RemoveContainer" containerID="238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452"
Nov 03 19:05:03.372733 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:03.372716    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452\": container with ID starting with 238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452 not found: ID does not exist" containerID="238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452"
Nov 03 19:05:03.372778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.372738    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452"} err="failed to get container status \"238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452\": rpc error: code = NotFound desc = could not find container \"238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452\": container with ID starting with 238d3ef7f63c390d78e67e6ffe8671f8846eee7b7c6f24b06e70ac4e5342a452 not found: ID does not exist"
Nov 03 19:05:03.588586 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.588511    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"]
Nov 03 19:05:03.613340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:03.613310    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-30b92815-openshift-default-746cd4b59c-9tpf6"]
Nov 03 19:05:04.995270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:04.995237    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" path="/var/lib/kubelet/pods/6a355f4a-4098-4c2d-94eb-5b1d1577e3a4/volumes"
Nov 03 19:05:17.575452 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:17.575380    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:05:30.939015 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.938982    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"]
Nov 03 19:05:30.939842 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939821    4489 cpu_manager.go:414] "containerMap: removing container" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy"
Nov 03 19:05:30.939899 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939847    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy"
Nov 03 19:05:30.939899 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939890    4489 cpu_manager.go:414] "containerMap: removing container" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy"
Nov 03 19:05:30.939963 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939899    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy"
Nov 03 19:05:30.939963 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939922    4489 cpu_manager.go:414] "containerMap: removing container" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy"
Nov 03 19:05:30.939963 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939929    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy"
Nov 03 19:05:30.939963 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939939    4489 cpu_manager.go:414] "containerMap: removing container" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy"
Nov 03 19:05:30.939963 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.939946    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy"
Nov 03 19:05:30.940108 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.940085    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy"
Nov 03 19:05:30.940154 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.940107    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy"
Nov 03 19:05:30.940154 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.940118    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1477b2-7e81-4df4-b942-ca04c61934aa" containerName="istio-proxy"
Nov 03 19:05:30.942638 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.942621    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:30.966342 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.966318    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:30.966427 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.966349    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-openshift-service-ca\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:30.966503 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.966489    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzt9\" (UniqueName: \"kubernetes.io/projected/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-kube-api-access-wmzt9\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:30.966563 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.966549    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-metrics-certs\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:30.998538 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:30.998516    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"]
Nov 03 19:05:31.046837 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.046812    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-82cd41ab-serving-cert\""
Nov 03 19:05:31.067891 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.067857    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzt9\" (UniqueName: \"kubernetes.io/projected/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-kube-api-access-wmzt9\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.068056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.068012    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-metrics-certs\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.068311 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.068282    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.068384 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.068319    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-openshift-service-ca\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.068987 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:31.068960    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-serving-cert: secret "maas-api-e2e-migrate-82cd41ab-serving-cert" not found
Nov 03 19:05:31.069114 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:31.069039    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls podName:bc2c0f66-9c84-4790-a65a-34ed49bc9ca2 nodeName:}" failed. No retries permitted until 2026-11-03 19:05:31.569018518 +0000 UTC m=+1660.717713084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls") pod "maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9" (UID: "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2") : secret "maas-api-e2e-migrate-82cd41ab-serving-cert" not found
Nov 03 19:05:31.069240 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.069208    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-openshift-service-ca\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.079643 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.079616    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-metrics-certs\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.131782 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.131710    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzt9\" (UniqueName: \"kubernetes.io/projected/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-kube-api-access-wmzt9\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.587853 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.587817    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.590366 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.590341    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:31.854204 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:31.854112    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:32.226562 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:05:32.226533    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2c0f66_9c84_4790_a65a_34ed49bc9ca2.slice/crio-4383d5f364720ed87be90271f527870df652221df72a3f091ed3a4f78738f3cd WatchSource:0}: Error finding container 4383d5f364720ed87be90271f527870df652221df72a3f091ed3a4f78738f3cd: Status 404 returned error can't find the container with id 4383d5f364720ed87be90271f527870df652221df72a3f091ed3a4f78738f3cd
Nov 03 19:05:32.228422 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.228403    4489 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:05:32.231178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.231132    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"]
Nov 03 19:05:32.514881 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.514793    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"]
Nov 03 19:05:32.515470 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.515455    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a355f4a-4098-4c2d-94eb-5b1d1577e3a4" containerName="istio-proxy"
Nov 03 19:05:32.518486 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.518467    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.584501 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.584468    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9" event={"ID":"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2","Type":"ContainerStarted","Data":"4383d5f364720ed87be90271f527870df652221df72a3f091ed3a4f78738f3cd"}
Nov 03 19:05:32.608367 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608343    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608491 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608382    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608491 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608420    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608581 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608532    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8bx\" (UniqueName: \"kubernetes.io/projected/6ddb111d-2ea5-43a9-8362-1122f85f28ac-kube-api-access-vx8bx\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608619 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608591    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608742 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608724    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608792 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608780    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608828 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608805    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.608871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.608847    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.615115 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.615088    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"]
Nov 03 19:05:32.710157 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710116    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710351 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710197    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8bx\" (UniqueName: \"kubernetes.io/projected/6ddb111d-2ea5-43a9-8362-1122f85f28ac-kube-api-access-vx8bx\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710351 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710262    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710351 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710340    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710517 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710489    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710517 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710499    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710556    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710674 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710630    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710674 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710633    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710817 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710800    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710859 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710837    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.710921 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.710858    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.711072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.711051    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.711427 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.711406    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.712934 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.712909    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.713056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.713044    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.774126 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.774065    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ddb111d-2ea5-43a9-8362-1122f85f28ac-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.774281 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.774124    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8bx\" (UniqueName: \"kubernetes.io/projected/6ddb111d-2ea5-43a9-8362-1122f85f28ac-kube-api-access-vx8bx\") pod \"maas-default-gateway-openshift-default-7854949c9-v7mn2\" (UID: \"6ddb111d-2ea5-43a9-8362-1122f85f28ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:32.833511 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:32.833475    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:33.256777 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:05:33.256742    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddb111d_2ea5_43a9_8362_1122f85f28ac.slice/crio-d7cea74c25ec5463a816389f23897db912809573cea9469cac44960d6835e514 WatchSource:0}: Error finding container d7cea74c25ec5463a816389f23897db912809573cea9469cac44960d6835e514: Status 404 returned error can't find the container with id d7cea74c25ec5463a816389f23897db912809573cea9469cac44960d6835e514
Nov 03 19:05:33.259677 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.259643    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:05:33.259797 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.259720    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:05:33.259797 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.259754    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:05:33.261068 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.261048    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"]
Nov 03 19:05:33.592161 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.592056    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" event={"ID":"6ddb111d-2ea5-43a9-8362-1122f85f28ac","Type":"ContainerStarted","Data":"4230ac8039bb4342f3648bd4c1bee983cda10bf8ccedfab4cb4b0c75daad9c56"}
Nov 03 19:05:33.592161 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.592114    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" event={"ID":"6ddb111d-2ea5-43a9-8362-1122f85f28ac","Type":"ContainerStarted","Data":"d7cea74c25ec5463a816389f23897db912809573cea9469cac44960d6835e514"}
Nov 03 19:05:33.593800 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.593770    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9" event={"ID":"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2","Type":"ContainerStarted","Data":"03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c"}
Nov 03 19:05:33.593952 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.593843    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:33.718599 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.718552    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podStartSLOduration=1.71853789 podStartE2EDuration="1.71853789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:33.717036695 +0000 UTC m=+1662.865731266" watchObservedRunningTime="2026-11-03 19:05:33.71853789 +0000 UTC m=+1662.867232455"
Nov 03 19:05:33.834117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.834084    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:33.841887 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:33.841826    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9" podStartSLOduration=3.016889986 podStartE2EDuration="3.841811588s" totalImagesPullingTime="824.921602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:33.838830524 +0000 UTC m=+1662.987525095" watchObservedRunningTime="2026-11-03 19:05:33.841811588 +0000 UTC m=+1662.990506169"
Nov 03 19:05:34.834915 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:34.834882    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:05:34.835354 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:34.834938    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 19:05:35.834658 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:35.834624    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:05:35.834836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:35.834681    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 19:05:36.834080 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:36.834046    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:05:36.834533 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:36.834101    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 19:05:37.834326 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:37.834293    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:05:37.834716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:37.834347    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 19:05:38.834093 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:38.834057    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:05:38.834267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:38.834111    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 19:05:39.605876 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:39.605839    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:05:39.833892 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:39.833860    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:05:39.834081 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:39.833918    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 19:05:40.432671 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.432641    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"]
Nov 03 19:05:40.435939 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.435918    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.491670 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.491630    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-9e277a-serving-cert\""
Nov 03 19:05:40.494901 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.494876    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"]
Nov 03 19:05:40.500640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.500618    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.500761 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.500677    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5707bcc4-3660-4c82-9bf5-c2858973132f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.500833 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.500782    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxsxc\" (UniqueName: \"kubernetes.io/projected/5707bcc4-3660-4c82-9bf5-c2858973132f-kube-api-access-rxsxc\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.500889 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.500872    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.602244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.602211    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.602428 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.602274    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5707bcc4-3660-4c82-9bf5-c2858973132f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.602428 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.602301    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxsxc\" (UniqueName: \"kubernetes.io/projected/5707bcc4-3660-4c82-9bf5-c2858973132f-kube-api-access-rxsxc\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.602428 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.602342    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.602428 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:40.602355    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-serving-cert: secret "maas-api-e2e-shared-a-w4-9e277a-serving-cert" not found
Nov 03 19:05:40.602428 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:40.602417    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls podName:5707bcc4-3660-4c82-9bf5-c2858973132f nodeName:}" failed. No retries permitted until 2026-11-03 19:05:41.102402578 +0000 UTC m=+1670.251097127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls") pod "maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9" (UID: "5707bcc4-3660-4c82-9bf5-c2858973132f") : secret "maas-api-e2e-shared-a-w4-9e277a-serving-cert" not found
Nov 03 19:05:40.603210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.603185    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5707bcc4-3660-4c82-9bf5-c2858973132f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.605041 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.605018    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.703534 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.703455    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxsxc\" (UniqueName: \"kubernetes.io/projected/5707bcc4-3660-4c82-9bf5-c2858973132f-kube-api-access-rxsxc\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:40.837085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.836986    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:05:40.837085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:40.837067    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 19:05:41.123840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:41.123750    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:41.126502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:41.126478    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:41.350728 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:41.350691    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:41.748965 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:05:41.748928    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5707bcc4_3660_4c82_9bf5_c2858973132f.slice/crio-f806e613f6c3e63fb6e93426327cac397d2cb6e18d8ef0c4ce4d9cd024b33857 WatchSource:0}: Error finding container f806e613f6c3e63fb6e93426327cac397d2cb6e18d8ef0c4ce4d9cd024b33857: Status 404 returned error can't find the container with id f806e613f6c3e63fb6e93426327cac397d2cb6e18d8ef0c4ce4d9cd024b33857
Nov 03 19:05:41.753088 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:41.753064    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"]
Nov 03 19:05:42.052729 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:42.052643    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:42.053046 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:42.053008    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:42.054046 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:42.054015    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2"
Nov 03 19:05:42.662468 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:42.662434    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9" event={"ID":"5707bcc4-3660-4c82-9bf5-c2858973132f","Type":"ContainerStarted","Data":"f806e613f6c3e63fb6e93426327cac397d2cb6e18d8ef0c4ce4d9cd024b33857"}
Nov 03 19:05:43.670396 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:43.670366    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9" event={"ID":"5707bcc4-3660-4c82-9bf5-c2858973132f","Type":"ContainerStarted","Data":"4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb"}
Nov 03 19:05:43.670799 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:43.670424    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:43.795536 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:43.795474    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9" podStartSLOduration=2.938494942 podStartE2EDuration="3.79545685s" totalImagesPullingTime="856.961908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:43.792344524 +0000 UTC m=+1672.941039095" watchObservedRunningTime="2026-11-03 19:05:43.79545685 +0000 UTC m=+1672.944151422"
Nov 03 19:05:47.533299 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:47.533256    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:05:47.639210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.639175    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"]
Nov 03 19:05:47.642288 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.642273    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.757670 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.757638    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"]
Nov 03 19:05:47.825443 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825358    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.825586 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825479    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.825586 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825565    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.825693 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825600    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.825794 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825775    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.825864 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825809    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtllk\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-kube-api-access-wtllk\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.825864 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825843    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.825949 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825929    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.826007 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.825993    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927435 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927403    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927435 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927441    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtllk\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-kube-api-access-wtllk\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927459    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927487    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927524    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927600    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927666    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927761    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927797    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.927925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927848    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.928086 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.927986    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.928086 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.928042    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.928370 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.928350    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.928417 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.928388    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.929890 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.929862    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.930194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.930175    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.991689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.991664    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:47.991793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:47.991735    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtllk\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-kube-api-access-wtllk\") pod \"maas-default-gateway-openshift-default-7854949c9-m4ph9\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:48.255316 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:48.255284    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:48.679556 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:05:48.679520    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0683b2c2_7569_4ab3_80d9_340c7b50a7ab.slice/crio-6ef6ffc5a1ed27743d86aee27f5af8ceba8a79983a180600cfd361c147ffdf8e WatchSource:0}: Error finding container 6ef6ffc5a1ed27743d86aee27f5af8ceba8a79983a180600cfd361c147ffdf8e: Status 404 returned error can't find the container with id 6ef6ffc5a1ed27743d86aee27f5af8ceba8a79983a180600cfd361c147ffdf8e
Nov 03 19:05:48.682393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:48.682359    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:05:48.682508 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:48.682431    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:05:48.682508 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:48.682467    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:05:48.684365 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:48.684343    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"]
Nov 03 19:05:48.714423 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:48.714395    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" event={"ID":"0683b2c2-7569-4ab3-80d9-340c7b50a7ab","Type":"ContainerStarted","Data":"6ef6ffc5a1ed27743d86aee27f5af8ceba8a79983a180600cfd361c147ffdf8e"}
Nov 03 19:05:49.682977 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.682949    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:05:49.722014 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.721977    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" event={"ID":"0683b2c2-7569-4ab3-80d9-340c7b50a7ab","Type":"ContainerStarted","Data":"c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08"}
Nov 03 19:05:49.931875 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.931820    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"]
Nov 03 19:05:49.932570 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.932529    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podStartSLOduration=2.932514909 podStartE2EDuration="2.932514909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:49.931374428 +0000 UTC m=+1679.080068999" watchObservedRunningTime="2026-11-03 19:05:49.932514909 +0000 UTC m=+1679.081209479"
Nov 03 19:05:49.935059 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.935006    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:49.978953 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.978922    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:49.979165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.979060    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e603e9aa-25c7-4336-b712-03b1209fa0c3-openshift-service-ca\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:49.979324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.979306    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpvh\" (UniqueName: \"kubernetes.io/projected/e603e9aa-25c7-4336-b712-03b1209fa0c3-kube-api-access-zvpvh\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:49.979399 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.979353    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-metrics-certs\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:49.989974 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.989947    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6765e083-serving-cert\""
Nov 03 19:05:49.992087 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:49.992063    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"]
Nov 03 19:05:50.081213 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.081178    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpvh\" (UniqueName: \"kubernetes.io/projected/e603e9aa-25c7-4336-b712-03b1209fa0c3-kube-api-access-zvpvh\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.081501 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.081481    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-metrics-certs\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.081782 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.081766    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.081991 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.081974    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e603e9aa-25c7-4336-b712-03b1209fa0c3-openshift-service-ca\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.085054 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:50.085024    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-serving-cert: secret "maas-api-e2e-ait-6765e083-serving-cert" not found
Nov 03 19:05:50.086293 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.085933    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e603e9aa-25c7-4336-b712-03b1209fa0c3-openshift-service-ca\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.086293 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.085990    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-metrics-certs\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.086463 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:05:50.086327    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls podName:e603e9aa-25c7-4336-b712-03b1209fa0c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:05:50.586243732 +0000 UTC m=+1679.734938284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls") pod "maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf" (UID: "e603e9aa-25c7-4336-b712-03b1209fa0c3") : secret "maas-api-e2e-ait-6765e083-serving-cert" not found
Nov 03 19:05:50.146366 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.146338    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpvh\" (UniqueName: \"kubernetes.io/projected/e603e9aa-25c7-4336-b712-03b1209fa0c3-kube-api-access-zvpvh\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.259072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.258586    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:50.259072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.258707    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:05:50.259072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.258753    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:05:50.604043 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.603958    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.606495 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.606471    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls\") pod \"maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:50.863222 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:50.848379    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:51.248468 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:05:51.248423    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode603e9aa_25c7_4336_b712_03b1209fa0c3.slice/crio-54bebdef76ee50afc447df173cd6351776876b8eb234bbe1fe44f75552eb4a54 WatchSource:0}: Error finding container 54bebdef76ee50afc447df173cd6351776876b8eb234bbe1fe44f75552eb4a54: Status 404 returned error can't find the container with id 54bebdef76ee50afc447df173cd6351776876b8eb234bbe1fe44f75552eb4a54
Nov 03 19:05:51.278225 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:51.270230    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:05:51.278388 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:51.278279    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:05:51.279501 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:51.279471    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"]
Nov 03 19:05:51.735077 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:51.735042    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf" event={"ID":"e603e9aa-25c7-4336-b712-03b1209fa0c3","Type":"ContainerStarted","Data":"54bebdef76ee50afc447df173cd6351776876b8eb234bbe1fe44f75552eb4a54"}
Nov 03 19:05:52.263284 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:52.263249    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:05:52.263722 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:52.263315    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:05:52.743619 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:52.743580    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf" event={"ID":"e603e9aa-25c7-4336-b712-03b1209fa0c3","Type":"ContainerStarted","Data":"c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997"}
Nov 03 19:05:52.743828 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:52.743684    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:05:52.872370 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:52.872303    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf" podStartSLOduration=2.995110369 podStartE2EDuration="3.872282141s" totalImagesPullingTime="877.171772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:52.869568575 +0000 UTC m=+1682.018263148" watchObservedRunningTime="2026-11-03 19:05:52.872282141 +0000 UTC m=+1682.020976712"
Nov 03 19:05:53.259618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:53.259584    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:05:53.259902 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:53.259853    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:05:54.256504 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:54.256467    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:05:54.257124 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:54.257092    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:05:55.256462 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:55.256425    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:05:55.256666 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:55.256501    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:05:56.255556 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:56.255521    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:05:56.255724 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:56.255578    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:05:57.326387 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:57.326355    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:57.326799 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:57.326755    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:57.327520 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:57.327495    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:05:58.774620 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:05:58.774593    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:06:17.532952 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:06:17.532914    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:06:22.742158 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:22.742110    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"]
Nov 03 19:06:22.742581 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:22.742342    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy" containerID="cri-o://cf7c939d59d2266d729de4725c74bea4ec3ceb58e20eea860d4630b8eab2830a" gracePeriod=30
Nov 03 19:06:28.028936 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.028863    4489 generic.go:356] "Generic (PLEG): container finished" podID="436255e1-65db-44db-b906-814953040195" containerID="cf7c939d59d2266d729de4725c74bea4ec3ceb58e20eea860d4630b8eab2830a" exitCode=0
Nov 03 19:06:28.029318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.028940    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" event={"ID":"436255e1-65db-44db-b906-814953040195","Type":"ContainerDied","Data":"cf7c939d59d2266d729de4725c74bea4ec3ceb58e20eea860d4630b8eab2830a"}
Nov 03 19:06:28.068476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.068455    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:06:28.163659 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163625    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-kube-api-access-2w2cl\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-kube-api-access-2w2cl\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.163792 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163704    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-data\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.163792 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163735    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-istio-token\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-istio-token\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.163868 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163813    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/436255e1-65db-44db-b906-814953040195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436255e1-65db-44db-b906-814953040195-istio-podinfo\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.163868 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163849    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436255e1-65db-44db-b906-814953040195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436255e1-65db-44db-b906-814953040195-istiod-ca-cert\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.163960 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163873    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-certs\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.163960 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163899    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-socket\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.163960 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.163950    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-credential-socket\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.164116 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164005    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-data" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:06:28.164116 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164042    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-envoy\") pod \"436255e1-65db-44db-b906-814953040195\" (UID: \"436255e1-65db-44db-b906-814953040195\") "
Nov 03 19:06:28.164268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164207    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-socket" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:06:28.164360 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164327    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-credential-socket" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:06:28.164482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164346    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436255e1-65db-44db-b906-814953040195-istiod-ca-cert" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:06:28.164482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164397    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-certs" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:06:28.164781 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164760    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436255e1-65db-44db-b906-814953040195-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.164781 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164781    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.164915 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164789    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.164915 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164799    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.164915 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.164807    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.166164 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.166129    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-istio-token" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:06:28.166525 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.166508    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-envoy" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:06:28.166650 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.166630    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/436255e1-65db-44db-b906-814953040195-istio-podinfo" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:06:28.166717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.166644    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-kube-api-access-2w2cl" pod "436255e1-65db-44db-b906-814953040195" (UID: "436255e1-65db-44db-b906-814953040195"). InnerVolumeSpecName "kube-api-access-2w2cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:06:28.266947 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.266911    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436255e1-65db-44db-b906-814953040195-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.266947 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.266947    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w2cl\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-kube-api-access-2w2cl\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.267232 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.266964    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436255e1-65db-44db-b906-814953040195-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:28.267232 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:28.266978    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436255e1-65db-44db-b906-814953040195-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:29.035644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:29.035610    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5" event={"ID":"436255e1-65db-44db-b906-814953040195","Type":"ContainerDied","Data":"a7bcf47e216771370c15688a7c4a80102c75cbfee5340e0e4551c3e5c4e59f7f"}
Nov 03 19:06:29.036077 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:29.035659    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"
Nov 03 19:06:29.036077 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:29.035667    4489 scope.go:122] "RemoveContainer" containerID="cf7c939d59d2266d729de4725c74bea4ec3ceb58e20eea860d4630b8eab2830a"
Nov 03 19:06:29.263514 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:29.263437    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"]
Nov 03 19:06:29.314044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:29.314000    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-6bjb5"]
Nov 03 19:06:30.996383 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:30.996345    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436255e1-65db-44db-b906-814953040195" path="/var/lib/kubelet/pods/436255e1-65db-44db-b906-814953040195/volumes"
Nov 03 19:06:47.532100 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:06:47.532061    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:06:53.900687 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.900648    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"]
Nov 03 19:06:53.901630 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.901608    4489 cpu_manager.go:414] "containerMap: removing container" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy"
Nov 03 19:06:53.901701 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.901635    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy"
Nov 03 19:06:53.901796 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.901782    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="436255e1-65db-44db-b906-814953040195" containerName="istio-proxy"
Nov 03 19:06:53.905933 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.905909    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:53.952674 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.952608    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73eaba1-ec4e-4348-a45c-b364300a95a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:53.953306 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.953268    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjdj\" (UniqueName: \"kubernetes.io/projected/c73eaba1-ec4e-4348-a45c-b364300a95a3-kube-api-access-vtjdj\") pod \"payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:53.969151 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:53.969093    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"]
Nov 03 19:06:54.012170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.010057    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-58ac09-dockercfg-b8sz2\""
Nov 03 19:06:54.012170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.010170    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-58ac09\""
Nov 03 19:06:54.057382 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.057341    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73eaba1-ec4e-4348-a45c-b364300a95a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:54.057651 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.057579    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjdj\" (UniqueName: \"kubernetes.io/projected/c73eaba1-ec4e-4348-a45c-b364300a95a3-kube-api-access-vtjdj\") pod \"payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:54.059273 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.059236    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73eaba1-ec4e-4348-a45c-b364300a95a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:54.105177 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.104558    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"]
Nov 03 19:06:54.110656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.109299    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.125268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.125045    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjdj\" (UniqueName: \"kubernetes.io/projected/c73eaba1-ec4e-4348-a45c-b364300a95a3-kube-api-access-vtjdj\") pod \"payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:54.169458 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.169382    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe49d37-6d34-49a2-8348-8dad98fd6883-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.169877 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.169849    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzjx\" (UniqueName: \"kubernetes.io/projected/fbe49d37-6d34-49a2-8348-8dad98fd6883-kube-api-access-cfzjx\") pod \"payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.176353 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.174210    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"]
Nov 03 19:06:54.221610 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.221584    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:54.272800 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.272763    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe49d37-6d34-49a2-8348-8dad98fd6883-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.273006 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.272978    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzjx\" (UniqueName: \"kubernetes.io/projected/fbe49d37-6d34-49a2-8348-8dad98fd6883-kube-api-access-cfzjx\") pod \"payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.273590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.273565    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe49d37-6d34-49a2-8348-8dad98fd6883-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.361801 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.347799    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzjx\" (UniqueName: \"kubernetes.io/projected/fbe49d37-6d34-49a2-8348-8dad98fd6883-kube-api-access-cfzjx\") pod \"payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.430251 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.427157    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:54.616418 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:06:54.616388    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73eaba1_ec4e_4348_a45c_b364300a95a3.slice/crio-57c1d399969cadefbd53e47f83d607ecfe9cefdde669c59e3f8586f9933d0fad WatchSource:0}: Error finding container 57c1d399969cadefbd53e47f83d607ecfe9cefdde669c59e3f8586f9933d0fad: Status 404 returned error can't find the container with id 57c1d399969cadefbd53e47f83d607ecfe9cefdde669c59e3f8586f9933d0fad
Nov 03 19:06:54.620726 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.620700    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"]
Nov 03 19:06:54.811178 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:06:54.811129    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe49d37_6d34_49a2_8348_8dad98fd6883.slice/crio-cbf51bbd59d171e18d39dde98c1427c4684aee14b319ebfe679024fefd0abdc5 WatchSource:0}: Error finding container cbf51bbd59d171e18d39dde98c1427c4684aee14b319ebfe679024fefd0abdc5: Status 404 returned error can't find the container with id cbf51bbd59d171e18d39dde98c1427c4684aee14b319ebfe679024fefd0abdc5
Nov 03 19:06:54.815965 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:54.815940    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"]
Nov 03 19:06:55.239595 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:55.239558    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9" event={"ID":"c73eaba1-ec4e-4348-a45c-b364300a95a3","Type":"ContainerStarted","Data":"98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d"}
Nov 03 19:06:55.239997 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:55.239611    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9" event={"ID":"c73eaba1-ec4e-4348-a45c-b364300a95a3","Type":"ContainerStarted","Data":"57c1d399969cadefbd53e47f83d607ecfe9cefdde669c59e3f8586f9933d0fad"}
Nov 03 19:06:55.239997 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:55.239642    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:06:55.241193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:55.241166    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78" event={"ID":"fbe49d37-6d34-49a2-8348-8dad98fd6883","Type":"ContainerStarted","Data":"3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2"}
Nov 03 19:06:55.241289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:55.241214    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78" event={"ID":"fbe49d37-6d34-49a2-8348-8dad98fd6883","Type":"ContainerStarted","Data":"cbf51bbd59d171e18d39dde98c1427c4684aee14b319ebfe679024fefd0abdc5"}
Nov 03 19:06:55.241289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:55.241267    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:06:55.364431 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:55.364377    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9" podStartSLOduration=2.364361179 podStartE2EDuration="2.364361179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:55.360826239 +0000 UTC m=+1744.509520810" watchObservedRunningTime="2026-11-03 19:06:55.364361179 +0000 UTC m=+1744.513055751"
Nov 03 19:06:59.134537 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.134482    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78" podStartSLOduration=5.13446582 podStartE2EDuration="5.13446582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:55.494733276 +0000 UTC m=+1744.643427877" watchObservedRunningTime="2026-11-03 19:06:59.13446582 +0000 UTC m=+1748.283160393"
Nov 03 19:06:59.137587 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.137561    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"]
Nov 03 19:06:59.142648 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.142632    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.196775 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.196743    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"]
Nov 03 19:06:59.244972 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.244942    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-770e67-serving-cert\""
Nov 03 19:06:59.250662 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.250635    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.250806 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.250770    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.250852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.250829    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685be93-62a5-4ea2-a56d-5322a23e6113-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.250906 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.250868    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwlw\" (UniqueName: \"kubernetes.io/projected/2685be93-62a5-4ea2-a56d-5322a23e6113-kube-api-access-vzwlw\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.353039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.352996    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.353266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.353104    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.353266 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:06:59.353179    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-serving-cert: secret "maas-api-e2e-shared-a-w2-770e67-serving-cert" not found
Nov 03 19:06:59.353266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.353214    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685be93-62a5-4ea2-a56d-5322a23e6113-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.353266 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:06:59.353247    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls podName:2685be93-62a5-4ea2-a56d-5322a23e6113 nodeName:}" failed. No retries permitted until 2026-11-03 19:06:59.853228973 +0000 UTC m=+1749.001923538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls") pod "maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n" (UID: "2685be93-62a5-4ea2-a56d-5322a23e6113") : secret "maas-api-e2e-shared-a-w2-770e67-serving-cert" not found
Nov 03 19:06:59.353502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.353328    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwlw\" (UniqueName: \"kubernetes.io/projected/2685be93-62a5-4ea2-a56d-5322a23e6113-kube-api-access-vzwlw\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.354016 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.353934    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685be93-62a5-4ea2-a56d-5322a23e6113-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.355802 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.355784    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.421516 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.421472    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwlw\" (UniqueName: \"kubernetes.io/projected/2685be93-62a5-4ea2-a56d-5322a23e6113-kube-api-access-vzwlw\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.863338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.863259    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:06:59.865741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:06:59.865719    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:07:00.058249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:00.058217    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:07:00.430878 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:07:00.430835    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2685be93_62a5_4ea2_a56d_5322a23e6113.slice/crio-49d15502977b3261edcaeec03ccae38321171c0502b48111a6e5b7cc32ab76cc WatchSource:0}: Error finding container 49d15502977b3261edcaeec03ccae38321171c0502b48111a6e5b7cc32ab76cc: Status 404 returned error can't find the container with id 49d15502977b3261edcaeec03ccae38321171c0502b48111a6e5b7cc32ab76cc
Nov 03 19:07:00.435123 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:00.435098    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"]
Nov 03 19:07:01.282321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:01.282272    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n" event={"ID":"2685be93-62a5-4ea2-a56d-5322a23e6113","Type":"ContainerStarted","Data":"49d15502977b3261edcaeec03ccae38321171c0502b48111a6e5b7cc32ab76cc"}
Nov 03 19:07:02.290403 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:02.290367    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n" event={"ID":"2685be93-62a5-4ea2-a56d-5322a23e6113","Type":"ContainerStarted","Data":"d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121"}
Nov 03 19:07:02.290808 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:02.290436    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:07:02.428169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:02.423839    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n" podStartSLOduration=2.591953315 podStartE2EDuration="3.423818701s" totalImagesPullingTime="831.865386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:02.419727723 +0000 UTC m=+1751.568422295" watchObservedRunningTime="2026-11-03 19:07:02.423818701 +0000 UTC m=+1751.572513273"
Nov 03 19:07:06.250434 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:06.250409    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:07:06.250807 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:06.250487    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:07:08.303096 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:08.303024    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:07:17.533821 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:07:17.533780    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:07:22.645493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:22.645458    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"]
Nov 03 19:07:22.645954 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:22.645662    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" containerID="cri-o://3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb" gracePeriod=30
Nov 03 19:07:26.690499 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:26.690461    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:07:26.690899 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:26.690537    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:07:28.009569 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.009542    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 19:07:28.103481 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103451    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-data\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103489    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-kube-api-access-p687f\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103510    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c59663c-a417-4f26-8ebc-66e4f99e2103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c59663c-a417-4f26-8ebc-66e4f99e2103-istiod-ca-cert\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103538    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-podinfo\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103573    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-certs\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103623    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-credential-socket\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103890 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103712    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-envoy\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103890 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103789    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-socket\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.103890 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103800    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-data" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:28.103890 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103866    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-certs" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:28.104106 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103972    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-credential-socket" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:28.104106 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.103986    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c59663c-a417-4f26-8ebc-66e4f99e2103-istiod-ca-cert" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:28.104106 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.104008    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-token\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-token\") pod \"2c59663c-a417-4f26-8ebc-66e4f99e2103\" (UID: \"2c59663c-a417-4f26-8ebc-66e4f99e2103\") "
Nov 03 19:07:28.104106 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.104088    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-socket" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:28.105126 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.105097    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.105274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.105130    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.105274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.105164    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c59663c-a417-4f26-8ebc-66e4f99e2103-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.105274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.105179    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.105274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.105194    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.106053 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.106025    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-kube-api-access-p687f" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "kube-api-access-p687f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:28.106190 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.106129    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-podinfo" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:07:28.106260 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.106241    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-envoy" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:07:28.106492 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.106473    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-token" pod "2c59663c-a417-4f26-8ebc-66e4f99e2103" (UID: "2c59663c-a417-4f26-8ebc-66e4f99e2103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:28.206661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.206608    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.206661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.206641    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/2c59663c-a417-4f26-8ebc-66e4f99e2103-kube-api-access-p687f\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.206661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.206657    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.206882 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.206668    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c59663c-a417-4f26-8ebc-66e4f99e2103-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:28.464092 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.464058    4489 generic.go:356] "Generic (PLEG): container finished" podID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerID="3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb" exitCode=0
Nov 03 19:07:28.464324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.464106    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"
Nov 03 19:07:28.464324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.464158    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" event={"ID":"2c59663c-a417-4f26-8ebc-66e4f99e2103","Type":"ContainerDied","Data":"3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb"}
Nov 03 19:07:28.464324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.464218    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk" event={"ID":"2c59663c-a417-4f26-8ebc-66e4f99e2103","Type":"ContainerDied","Data":"305da3d9ac2ccff046901019951e5c6da8a5f6344bb8ae99aeabc73919a2ebe3"}
Nov 03 19:07:28.464324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.464246    4489 scope.go:122] "RemoveContainer" containerID="3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb"
Nov 03 19:07:28.475409 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.475390    4489 scope.go:122] "RemoveContainer" containerID="3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb"
Nov 03 19:07:28.475704 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:07:28.475686    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb\": container with ID starting with 3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb not found: ID does not exist" containerID="3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb"
Nov 03 19:07:28.475752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.475715    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb"} err="failed to get container status \"3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb\": rpc error: code = NotFound desc = could not find container \"3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb\": container with ID starting with 3b99fb951dac5ad82e231caab7728560b7c74915418c353cb43700f8965d3efb not found: ID does not exist"
Nov 03 19:07:28.636619 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.636589    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"]
Nov 03 19:07:28.640848 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:28.640827    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-cj9vk"]
Nov 03 19:07:29.005780 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:29.005742    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" path="/var/lib/kubelet/pods/2c59663c-a417-4f26-8ebc-66e4f99e2103/volumes"
Nov 03 19:07:32.784222 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:32.784180    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"]
Nov 03 19:07:32.784764 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:32.784481    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9" podUID="bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" containerName="maas-api" containerID="cri-o://03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c" gracePeriod=30
Nov 03 19:07:33.116997 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.116969    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:07:33.185166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.180594    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-openshift-service-ca\") pod \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") "
Nov 03 19:07:33.185166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.180660    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-metrics-certs\") pod \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") "
Nov 03 19:07:33.185166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.180748    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls\") pod \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") "
Nov 03 19:07:33.185166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.180929    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-kube-api-access-wmzt9\" (UniqueName: \"kubernetes.io/projected/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-kube-api-access-wmzt9\") pod \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\" (UID: \"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2\") "
Nov 03 19:07:33.185166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.184334    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-openshift-service-ca" pod "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" (UID: "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:33.186444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.186416    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls" pod "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" (UID: "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:33.190005 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.189973    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-metrics-certs" pod "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" (UID: "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:33.190243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.190126    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-kube-api-access-wmzt9" pod "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" (UID: "bc2c0f66-9c84-4790-a65a-34ed49bc9ca2"). InnerVolumeSpecName "kube-api-access-wmzt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:33.283959 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.283906    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmzt9\" (UniqueName: \"kubernetes.io/projected/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-kube-api-access-wmzt9\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:33.283959 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.283950    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:33.284280 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.283968    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:33.284280 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.283983    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:33.499175 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.499097    4489 generic.go:356] "Generic (PLEG): container finished" podID="bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" containerID="03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c" exitCode=0
Nov 03 19:07:33.499379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.499181    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"
Nov 03 19:07:33.499379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.499223    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9" event={"ID":"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2","Type":"ContainerDied","Data":"03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c"}
Nov 03 19:07:33.499379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.499301    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9" event={"ID":"bc2c0f66-9c84-4790-a65a-34ed49bc9ca2","Type":"ContainerDied","Data":"4383d5f364720ed87be90271f527870df652221df72a3f091ed3a4f78738f3cd"}
Nov 03 19:07:33.499379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.499345    4489 scope.go:122] "RemoveContainer" containerID="03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c"
Nov 03 19:07:33.511970 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.511953    4489 scope.go:122] "RemoveContainer" containerID="03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c"
Nov 03 19:07:33.512273 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:07:33.512255    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c\": container with ID starting with 03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c not found: ID does not exist" containerID="03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c"
Nov 03 19:07:33.512351 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.512281    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c"} err="failed to get container status \"03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c\": rpc error: code = NotFound desc = could not find container \"03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c\": container with ID starting with 03cac0c594e2f342ef6a4172faffad4b840e411803c1fde5946c85fbf94b508c not found: ID does not exist"
Nov 03 19:07:33.671918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.671877    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"]
Nov 03 19:07:33.675274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:33.675252    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-82cd41ab-6bb59b5ff9-gvnc9"]
Nov 03 19:07:34.997698 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:34.997665    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" path="/var/lib/kubelet/pods/bc2c0f66-9c84-4790-a65a-34ed49bc9ca2/volumes"
Nov 03 19:07:36.479303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.479256    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"]
Nov 03 19:07:36.480117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.480094    4489 cpu_manager.go:414] "containerMap: removing container" podUID="bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" containerName="maas-api"
Nov 03 19:07:36.480117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.480115    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" containerName="maas-api"
Nov 03 19:07:36.480358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.480201    4489 cpu_manager.go:414] "containerMap: removing container" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy"
Nov 03 19:07:36.480358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.480211    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy"
Nov 03 19:07:36.480358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.480358    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2c0f66-9c84-4790-a65a-34ed49bc9ca2" containerName="maas-api"
Nov 03 19:07:36.480504 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.480376    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c59663c-a417-4f26-8ebc-66e4f99e2103" containerName="istio-proxy"
Nov 03 19:07:36.487793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.487759    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.558203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.549831    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-bf3e1c-openshift-default-dockercfg-v6m9k\""
Nov 03 19:07:36.593672 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.593633    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-token\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.593951 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.593927    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-envoy\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.594108 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.593995    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-certs\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.594108 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.594022    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-data\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.594108 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.594099    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkpw\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-kube-api-access-zvkpw\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.594317 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.594183    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-credential-socket\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.594368 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.594346    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-podinfo\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.594478 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.594462    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319b6ca2-9d70-4ac4-b521-c5868508fb51-istiod-ca-cert\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.594535 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.594493    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-socket\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.598812 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.598787    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"]
Nov 03 19:07:36.695891 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.695855    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-podinfo\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.695952    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319b6ca2-9d70-4ac4-b521-c5868508fb51-istiod-ca-cert\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.695972    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-socket\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.696006    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-token\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.696033    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-envoy\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.696073    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-certs\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.696095    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-data\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.696185    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkpw\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-kube-api-access-zvkpw\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.696221    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-credential-socket\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.697012    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-socket\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.697234    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319b6ca2-9d70-4ac4-b521-c5868508fb51-istiod-ca-cert\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.697359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.697300    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-certs\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.698001 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.697570    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-data\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.698001 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.697763    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-credential-socket\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.699358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.699330    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-envoy\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.699482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.699459    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-podinfo\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.760880 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.760802    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-token\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.762426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.762395    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkpw\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-kube-api-access-zvkpw\") pod \"e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") " pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:36.821700 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:36.821664    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:37.306349 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:07:37.306312    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319b6ca2_9d70_4ac4_b521_c5868508fb51.slice/crio-bdd8a511fab21de0c167c89489ae065b2d5eab3dcd961be320f9148da94b9a23 WatchSource:0}: Error finding container bdd8a511fab21de0c167c89489ae065b2d5eab3dcd961be320f9148da94b9a23: Status 404 returned error can't find the container with id bdd8a511fab21de0c167c89489ae065b2d5eab3dcd961be320f9148da94b9a23
Nov 03 19:07:37.310819 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.310791    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"]
Nov 03 19:07:37.316015 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.315984    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:07:37.316109 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.316046    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:07:37.316109 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.316078    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:07:37.598561 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.598465    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" event={"ID":"319b6ca2-9d70-4ac4-b521-c5868508fb51","Type":"ContainerStarted","Data":"e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b"}
Nov 03 19:07:37.598561 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.598535    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" event={"ID":"319b6ca2-9d70-4ac4-b521-c5868508fb51","Type":"ContainerStarted","Data":"bdd8a511fab21de0c167c89489ae065b2d5eab3dcd961be320f9148da94b9a23"}
Nov 03 19:07:37.730136 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.730091    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podStartSLOduration=1.7300769599999999 podStartE2EDuration="1.73007696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:37.727076037 +0000 UTC m=+1786.875770631" watchObservedRunningTime="2026-11-03 19:07:37.73007696 +0000 UTC m=+1786.878771530"
Nov 03 19:07:37.822455 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:37.822426    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:38.828274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:38.828243    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:39.620545 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:39.620516    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:39.621416 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:39.621395    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:07:47.534087 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:07:47.534051    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:07:49.585199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:49.585162    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"]
Nov 03 19:07:49.585594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:49.585413    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf" podUID="e603e9aa-25c7-4336-b712-03b1209fa0c3" containerName="maas-api" containerID="cri-o://c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997" gracePeriod=30
Nov 03 19:07:49.929474 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:49.929445    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:07:50.114304 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.114272    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e603e9aa-25c7-4336-b712-03b1209fa0c3-kube-api-access-zvpvh\" (UniqueName: \"kubernetes.io/projected/e603e9aa-25c7-4336-b712-03b1209fa0c3-kube-api-access-zvpvh\") pod \"e603e9aa-25c7-4336-b712-03b1209fa0c3\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") "
Nov 03 19:07:50.114498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.114328    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-metrics-certs\") pod \"e603e9aa-25c7-4336-b712-03b1209fa0c3\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") "
Nov 03 19:07:50.114498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.114354    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e603e9aa-25c7-4336-b712-03b1209fa0c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e603e9aa-25c7-4336-b712-03b1209fa0c3-openshift-service-ca\") pod \"e603e9aa-25c7-4336-b712-03b1209fa0c3\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") "
Nov 03 19:07:50.114498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.114442    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls\") pod \"e603e9aa-25c7-4336-b712-03b1209fa0c3\" (UID: \"e603e9aa-25c7-4336-b712-03b1209fa0c3\") "
Nov 03 19:07:50.115034 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.115002    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e603e9aa-25c7-4336-b712-03b1209fa0c3-openshift-service-ca" pod "e603e9aa-25c7-4336-b712-03b1209fa0c3" (UID: "e603e9aa-25c7-4336-b712-03b1209fa0c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:50.115470 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.115449    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e603e9aa-25c7-4336-b712-03b1209fa0c3-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:50.117617 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.117566    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-metrics-certs" pod "e603e9aa-25c7-4336-b712-03b1209fa0c3" (UID: "e603e9aa-25c7-4336-b712-03b1209fa0c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:50.117886 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.117814    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls" pod "e603e9aa-25c7-4336-b712-03b1209fa0c3" (UID: "e603e9aa-25c7-4336-b712-03b1209fa0c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:50.117886 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.117838    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e603e9aa-25c7-4336-b712-03b1209fa0c3-kube-api-access-zvpvh" pod "e603e9aa-25c7-4336-b712-03b1209fa0c3" (UID: "e603e9aa-25c7-4336-b712-03b1209fa0c3"). InnerVolumeSpecName "kube-api-access-zvpvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:50.217593 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.217551    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvpvh\" (UniqueName: \"kubernetes.io/projected/e603e9aa-25c7-4336-b712-03b1209fa0c3-kube-api-access-zvpvh\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:50.217593 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.217594    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:50.217908 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.217609    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e603e9aa-25c7-4336-b712-03b1209fa0c3-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:50.715391 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.715348    4489 generic.go:356] "Generic (PLEG): container finished" podID="e603e9aa-25c7-4336-b712-03b1209fa0c3" containerID="c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997" exitCode=0
Nov 03 19:07:50.716249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.716220    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf" event={"ID":"e603e9aa-25c7-4336-b712-03b1209fa0c3","Type":"ContainerDied","Data":"c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997"}
Nov 03 19:07:50.716460 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.716441    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf" event={"ID":"e603e9aa-25c7-4336-b712-03b1209fa0c3","Type":"ContainerDied","Data":"54bebdef76ee50afc447df173cd6351776876b8eb234bbe1fe44f75552eb4a54"}
Nov 03 19:07:50.716590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.716574    4489 scope.go:122] "RemoveContainer" containerID="c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997"
Nov 03 19:07:50.716858 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.716841    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"
Nov 03 19:07:50.739512 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.739481    4489 scope.go:122] "RemoveContainer" containerID="c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997"
Nov 03 19:07:50.739851 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:07:50.739824    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997\": container with ID starting with c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997 not found: ID does not exist" containerID="c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997"
Nov 03 19:07:50.739947 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.739863    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997"} err="failed to get container status \"c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997\": rpc error: code = NotFound desc = could not find container \"c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997\": container with ID starting with c5c9b94e5d861b092ba10457b48ce4722646433fca579a1f08e0cd8f19eca997 not found: ID does not exist"
Nov 03 19:07:50.896550 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.896516    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"]
Nov 03 19:07:50.948693 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:50.948666    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6765e083-766d4fdc59-q7tjf"]
Nov 03 19:07:51.008401 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.008324    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e603e9aa-25c7-4336-b712-03b1209fa0c3" path="/var/lib/kubelet/pods/e603e9aa-25c7-4336-b712-03b1209fa0c3/volumes"
Nov 03 19:07:51.232283 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.232254    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"]
Nov 03 19:07:51.232848 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.232835    4489 cpu_manager.go:414] "containerMap: removing container" podUID="e603e9aa-25c7-4336-b712-03b1209fa0c3" containerName="maas-api"
Nov 03 19:07:51.232897 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.232849    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e603e9aa-25c7-4336-b712-03b1209fa0c3" containerName="maas-api"
Nov 03 19:07:51.232959 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.232950    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="e603e9aa-25c7-4336-b712-03b1209fa0c3" containerName="maas-api"
Nov 03 19:07:51.237562 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.237545    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.292576 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.292491    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-58ac09-serving-cert\""
Nov 03 19:07:51.296950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.296927    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"]
Nov 03 19:07:51.343814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.343784    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.344060 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.344034    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.344257 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.344237    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf2bg\" (UniqueName: \"kubernetes.io/projected/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-kube-api-access-cf2bg\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.344322 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.344269    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.445255 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.445221    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.445421 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.445404    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.445541 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.445521    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2bg\" (UniqueName: \"kubernetes.io/projected/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-kube-api-access-cf2bg\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.445577 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:07:51.445561    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-serving-cert: secret "maas-api-e2e-shared-b-w4-58ac09-serving-cert" not found
Nov 03 19:07:51.445645 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.445561    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.445708 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:07:51.445633    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls podName:b3ec524a-f2fa-4c3c-99c4-78fe30bfa457 nodeName:}" failed. No retries permitted until 2026-11-03 19:07:51.945612696 +0000 UTC m=+1801.094307258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls") pod "maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6" (UID: "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457") : secret "maas-api-e2e-shared-b-w4-58ac09-serving-cert" not found
Nov 03 19:07:51.446234 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.446196    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.447935 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.447913    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.513086 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.513049    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2bg\" (UniqueName: \"kubernetes.io/projected/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-kube-api-access-cf2bg\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.952917 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.952876    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:51.955541 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:51.955514    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:52.150355 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:52.150319    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:52.570267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:52.561517    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"]
Nov 03 19:07:52.741003 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:52.740966    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6" event={"ID":"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457","Type":"ContainerStarted","Data":"d4254749f2defe33ebbd8c999f146873256bd73c465ad96b169414a479424c0f"}
Nov 03 19:07:53.748695 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:53.748650    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6" event={"ID":"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457","Type":"ContainerStarted","Data":"424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03"}
Nov 03 19:07:53.749198 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:53.748865    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:07:53.929221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:53.905442    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6" podStartSLOduration=1.970685337 podStartE2EDuration="2.905419486s" totalImagesPullingTime="934.734149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:53.876856231 +0000 UTC m=+1803.025550827" watchObservedRunningTime="2026-11-03 19:07:53.905419486 +0000 UTC m=+1803.054114061"
Nov 03 19:07:59.764789 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:07:59.764754    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:08:17.532287 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:08:17.532252    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:08:47.597362 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:08:47.597327    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:09:17.612033 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:09:17.611991    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:09:21.668495 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:21.668415    4489 generic.go:356] "Generic (PLEG): container finished" podID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerID="e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b" exitCode=0
Nov 03 19:09:21.668495 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:21.668463    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" event={"ID":"319b6ca2-9d70-4ac4-b521-c5868508fb51","Type":"ContainerDied","Data":"e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b"}
Nov 03 19:09:21.669117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:21.668967    4489 scope.go:122] "RemoveContainer" containerID="e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b"
Nov 03 19:09:21.670783 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:21.670752    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:09:21.670847 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:21.670825    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:09:21.670886 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:21.670858    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:09:22.683758 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:22.683725    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" event={"ID":"319b6ca2-9d70-4ac4-b521-c5868508fb51","Type":"ContainerStarted","Data":"89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834"}
Nov 03 19:09:22.822326 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:22.822284    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:09:22.823803 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:22.823774    4489 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:09:22.823930 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:22.823827    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:09:23.822416 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:23.822379    4489 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:09:23.822838 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:23.822452    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:09:24.623653 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:24.623617    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:09:24.822203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:24.822168    4489 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:09:24.822379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:24.822223    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:09:25.822160 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:25.822104    4489 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:09:25.822677 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:25.822192    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:09:26.822725 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:26.822687    4489 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:09:26.823186 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:26.822761    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:09:27.822533 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:27.822494    4489 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:09:27.822753 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:27.822569    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:09:28.827081 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:28.827051    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:09:28.828180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:09:28.828133    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:09:47.532819 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:09:47.532785    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:10:09.023995 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:09.023886    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"]
Nov 03 19:10:09.024543 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:09.024236    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" containerID="cri-o://830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8" gracePeriod=30
Nov 03 19:10:09.141072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:09.141039    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"]
Nov 03 19:10:09.141362 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:09.141330    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy" containerID="cri-o://58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3" gracePeriod=30
Nov 03 19:10:13.518252 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:13.518212    4489 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:10:13.518690 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:13.518285    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:10:14.327043 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.327014    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:10:14.372111 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372068    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-data\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372317 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372298    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-envoy\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372402 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372369    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-podinfo\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372519 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372496    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-data" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.372519 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372510    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-credential-socket\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372553    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-token\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-token\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372634    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-socket\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372750 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372731    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-certs\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372809 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372792    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istiod-ca-cert\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.372868 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372845    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-kube-api-access-qfl45\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-kube-api-access-qfl45\") pod \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\" (UID: \"5373c6d5-a7f0-4dc7-9580-336354deb1cb\") "
Nov 03 19:10:14.373212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.372973    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-socket" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.373212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.373110    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-credential-socket" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.373212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.373176    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-certs" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.389814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.376575    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.389814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.376605    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.389814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.376631    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.389814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.376646    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.389814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.376804    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istiod-ca-cert" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:10:14.389814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.378417    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-token" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:10:14.389814 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.379443    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-podinfo" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:10:14.397304 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.397236    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-kube-api-access-qfl45" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "kube-api-access-qfl45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:10:14.399053 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.399018    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-envoy" pod "5373c6d5-a7f0-4dc7-9580-336354deb1cb" (UID: "5373c6d5-a7f0-4dc7-9580-336354deb1cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.415769 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.415746    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:10:14.477850 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.477810    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-certs\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.477915    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd596047-c356-4d71-b722-bc90cdbded25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd596047-c356-4d71-b722-bc90cdbded25-istiod-ca-cert\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.477951    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-envoy\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.477980    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-istio-token\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-istio-token\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478042    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-kube-api-access-rw4q9\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-kube-api-access-rw4q9\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478103    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd596047-c356-4d71-b722-bc90cdbded25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd596047-c356-4d71-b722-bc90cdbded25-istio-podinfo\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478109    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-certs" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.478253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478236    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-data\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478468 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478283    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-socket\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478468 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478392    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd596047-c356-4d71-b722-bc90cdbded25-istiod-ca-cert" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:10:14.478582 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478500    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-credential-socket\") pod \"dd596047-c356-4d71-b722-bc90cdbded25\" (UID: \"dd596047-c356-4d71-b722-bc90cdbded25\") "
Nov 03 19:10:14.478582 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478514    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-data" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.479260 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.478697    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-socket" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.479260 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479006    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-credential-socket" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.479470 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479420    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479470 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479446    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfl45\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-kube-api-access-qfl45\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479470 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479462    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479476    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479489    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479503    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd596047-c356-4d71-b722-bc90cdbded25-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479516    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479530    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5373c6d5-a7f0-4dc7-9580-336354deb1cb-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479544    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.479637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.479556    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.480538 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.480516    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-istio-token" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:10:14.480627 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.480558    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd596047-c356-4d71-b722-bc90cdbded25-istio-podinfo" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:10:14.480727 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.480712    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-kube-api-access-rw4q9" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "kube-api-access-rw4q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:10:14.480787 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.480772    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-envoy" pod "dd596047-c356-4d71-b722-bc90cdbded25" (UID: "dd596047-c356-4d71-b722-bc90cdbded25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:10:14.580788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.580692    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd596047-c356-4d71-b722-bc90cdbded25-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.580788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.580723    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.580788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.580734    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4q9\" (UniqueName: \"kubernetes.io/projected/dd596047-c356-4d71-b722-bc90cdbded25-kube-api-access-rw4q9\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:14.580788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:14.580745    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd596047-c356-4d71-b722-bc90cdbded25-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:10:15.089245 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.089214    4489 generic.go:356] "Generic (PLEG): container finished" podID="dd596047-c356-4d71-b722-bc90cdbded25" containerID="58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3" exitCode=0
Nov 03 19:10:15.089438 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.089267    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"
Nov 03 19:10:15.089438 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.089314    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" event={"ID":"dd596047-c356-4d71-b722-bc90cdbded25","Type":"ContainerDied","Data":"58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3"}
Nov 03 19:10:15.089438 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.089381    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j" event={"ID":"dd596047-c356-4d71-b722-bc90cdbded25","Type":"ContainerDied","Data":"08af116bf05ebbcd73f79a5a40efab8bfd881f32535343e07385d21a747a554e"}
Nov 03 19:10:15.089438 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.089420    4489 scope.go:122] "RemoveContainer" containerID="58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3"
Nov 03 19:10:15.090863 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.090841    4489 generic.go:356] "Generic (PLEG): container finished" podID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerID="830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8" exitCode=0
Nov 03 19:10:15.090971 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.090892    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" event={"ID":"5373c6d5-a7f0-4dc7-9580-336354deb1cb","Type":"ContainerDied","Data":"830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8"}
Nov 03 19:10:15.090971 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.090938    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"
Nov 03 19:10:15.090971 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.090944    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9" event={"ID":"5373c6d5-a7f0-4dc7-9580-336354deb1cb","Type":"ContainerDied","Data":"bb09dd5735ce077b732dee79fddea48fd022e71835513956558f635b68bdef49"}
Nov 03 19:10:15.100012 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.099992    4489 scope.go:122] "RemoveContainer" containerID="58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3"
Nov 03 19:10:15.100344 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:10:15.100316    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3\": container with ID starting with 58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3 not found: ID does not exist" containerID="58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3"
Nov 03 19:10:15.100445 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.100350    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3"} err="failed to get container status \"58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3\": rpc error: code = NotFound desc = could not find container \"58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3\": container with ID starting with 58bba20266594d9968f7eb4d5287ba7906bc94b8b3545ddefac94f91831b32b3 not found: ID does not exist"
Nov 03 19:10:15.100445 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.100373    4489 scope.go:122] "RemoveContainer" containerID="830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8"
Nov 03 19:10:15.110319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.110302    4489 scope.go:122] "RemoveContainer" containerID="830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8"
Nov 03 19:10:15.110604 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:10:15.110585    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8\": container with ID starting with 830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8 not found: ID does not exist" containerID="830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8"
Nov 03 19:10:15.110652 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.110618    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8"} err="failed to get container status \"830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8\": rpc error: code = NotFound desc = could not find container \"830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8\": container with ID starting with 830958726dcd79e1dcb8ccd051a333bca6ced312e588ff6273dbb4f168c652f8 not found: ID does not exist"
Nov 03 19:10:15.263422 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.263334    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"]
Nov 03 19:10:15.314419 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.314380    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-zfzp9"]
Nov 03 19:10:15.383534 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.383497    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"]
Nov 03 19:10:15.387218 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:15.387187    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-w296j"]
Nov 03 19:10:16.996529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:16.996497    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" path="/var/lib/kubelet/pods/5373c6d5-a7f0-4dc7-9580-336354deb1cb/volumes"
Nov 03 19:10:16.997022 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:10:16.997007    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd596047-c356-4d71-b722-bc90cdbded25" path="/var/lib/kubelet/pods/dd596047-c356-4d71-b722-bc90cdbded25/volumes"
Nov 03 19:10:17.535712 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:10:17.535676    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:10:47.653489 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:10:47.652589    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:11:17.536838 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:11:17.536792    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:11:28.853285 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.853246    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"]
Nov 03 19:11:28.854258 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.854234    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy"
Nov 03 19:11:28.854258 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.854257    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy"
Nov 03 19:11:28.854400 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.854310    4489 cpu_manager.go:414] "containerMap: removing container" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy"
Nov 03 19:11:28.854400 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.854318    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy"
Nov 03 19:11:28.854464 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.854452    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd596047-c356-4d71-b722-bc90cdbded25" containerName="istio-proxy"
Nov 03 19:11:28.854497 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.854470    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5373c6d5-a7f0-4dc7-9580-336354deb1cb" containerName="istio-proxy"
Nov 03 19:11:28.859013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.858982    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:28.915721 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.915669    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"]
Nov 03 19:11:28.959340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.959307    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4320e5\""
Nov 03 19:11:28.959592 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.959316    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4320e5-dockercfg-w22cs\""
Nov 03 19:11:28.979273 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:28.979232    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769d8b3-2246-4dca-8bb4-65f32f77f192-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.001454 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.001414    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrrl\" (UniqueName: \"kubernetes.io/projected/1769d8b3-2246-4dca-8bb4-65f32f77f192-kube-api-access-cwrrl\") pod \"payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.108556 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.107543    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrrl\" (UniqueName: \"kubernetes.io/projected/1769d8b3-2246-4dca-8bb4-65f32f77f192-kube-api-access-cwrrl\") pod \"payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.108556 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.107698    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769d8b3-2246-4dca-8bb4-65f32f77f192-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.108556 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.108473    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769d8b3-2246-4dca-8bb4-65f32f77f192-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.170812 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.170780    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrrl\" (UniqueName: \"kubernetes.io/projected/1769d8b3-2246-4dca-8bb4-65f32f77f192-kube-api-access-cwrrl\") pod \"payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.179714 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.179690    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.552225 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:11:29.552192    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1769d8b3_2246_4dca_8bb4_65f32f77f192.slice/crio-2989a1a0f1ce4f5d071f7e10d6217eac986c7c10000709f35dd412e3581a1a65 WatchSource:0}: Error finding container 2989a1a0f1ce4f5d071f7e10d6217eac986c7c10000709f35dd412e3581a1a65: Status 404 returned error can't find the container with id 2989a1a0f1ce4f5d071f7e10d6217eac986c7c10000709f35dd412e3581a1a65
Nov 03 19:11:29.558655 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.558630    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"]
Nov 03 19:11:29.756024 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.755981    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4" event={"ID":"1769d8b3-2246-4dca-8bb4-65f32f77f192","Type":"ContainerStarted","Data":"36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15"}
Nov 03 19:11:29.756251 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.756049    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:29.756251 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.756067    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4" event={"ID":"1769d8b3-2246-4dca-8bb4-65f32f77f192","Type":"ContainerStarted","Data":"2989a1a0f1ce4f5d071f7e10d6217eac986c7c10000709f35dd412e3581a1a65"}
Nov 03 19:11:29.879355 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:29.879258    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4" podStartSLOduration=1.879245112 podStartE2EDuration="1.879245112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:29.875314329 +0000 UTC m=+2019.024008903" watchObservedRunningTime="2026-11-03 19:11:29.879245112 +0000 UTC m=+2019.027939684"
Nov 03 19:11:40.765004 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:11:40.764918    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:11:47.535476 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:11:47.535432    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:12:09.132031 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:09.131994    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"]
Nov 03 19:12:09.132568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:09.132288    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" containerID="cri-o://0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa" gracePeriod=30
Nov 03 19:12:09.186209 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:09.186172    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"]
Nov 03 19:12:09.186463 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:09.186436    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" containerID="cri-o://65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea" gracePeriod=30
Nov 03 19:12:11.978818 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:11.978785    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"]
Nov 03 19:12:11.983199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:11.983180    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.067186 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.067129    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"]
Nov 03 19:12:12.114083 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.114057    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-97e020-openshift-default-dockercfg-htkrg\""
Nov 03 19:12:12.123224 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123194    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123376 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123250    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123376 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123329    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123376 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123361    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123511 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123392    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123550 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123500    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123643 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123624    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123798 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123782    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2vl\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-kube-api-access-9t2vl\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.123851 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.123808    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.225731 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.225676    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.225948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.225768    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.225948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.225802    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.225948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.225828    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.225948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.225865    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.225948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.225906    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226261 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.225970    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226261 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.226096    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226261 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.226208    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226410 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.226284    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226592 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.226567    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2vl\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-kube-api-access-9t2vl\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226658 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.226616    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226718 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.226668    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.226941 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.226922    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.228962 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.228901    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.229105 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.229079    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.290171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.290107    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2vl\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-kube-api-access-9t2vl\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.290331 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.290244    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.297178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.297137    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:12.651207 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.651100    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:12:12.651207 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.651170    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:12:12.730852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.730813    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"]
Nov 03 19:12:12.748819 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.748784    4489 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:12:12.748972 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.748866    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:12:12.758600 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:12:12.758572    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c674b2f_1db3_4a15_9e8f_aa148d40025e.slice/crio-a1287fb53de6a9a7f043cfb86faf537f2c6e552168bda60ba4aad7e1d408753f WatchSource:0}: Error finding container a1287fb53de6a9a7f043cfb86faf537f2c6e552168bda60ba4aad7e1d408753f: Status 404 returned error can't find the container with id a1287fb53de6a9a7f043cfb86faf537f2c6e552168bda60ba4aad7e1d408753f
Nov 03 19:12:12.761479 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.761448    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:12:12.761560 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.761520    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:12:12.761560 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:12.761549    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:12:13.047788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:13.047752    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" event={"ID":"0c674b2f-1db3-4a15-9e8f-aa148d40025e","Type":"ContainerStarted","Data":"30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3"}
Nov 03 19:12:13.048233 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:13.047804    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" event={"ID":"0c674b2f-1db3-4a15-9e8f-aa148d40025e","Type":"ContainerStarted","Data":"a1287fb53de6a9a7f043cfb86faf537f2c6e552168bda60ba4aad7e1d408753f"}
Nov 03 19:12:13.174295 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:13.174240    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podStartSLOduration=2.174221779 podStartE2EDuration="2.174221779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:13.170060111 +0000 UTC m=+2062.318754676" watchObservedRunningTime="2026-11-03 19:12:13.174221779 +0000 UTC m=+2062.322916354"
Nov 03 19:12:13.297618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:13.297582    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:13.299267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:13.299201    4489 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:12:13.299385 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:13.299259    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:12:14.298605 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.298571    4489 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:12:14.298959 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.298632    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:12:14.541232 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.541204    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:12:14.598462 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.598437    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:12:14.686096 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686066    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-socket\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686295 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686183    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-kube-api-access-9s2lt\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-kube-api-access-9s2lt\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686295 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686214    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-podinfo\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.686295 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686248    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-credential-socket\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686326    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-podinfo\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686377    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-kube-api-access-bbhdb\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-kube-api-access-bbhdb\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.686430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686386    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-socket" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.686430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686405    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-token\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-token\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686573    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-data\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686698    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-data\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.686716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686704    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-credential-socket" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.686822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686741    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-envoy\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.686822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686774    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istiod-ca-cert\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686797    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istiod-ca-cert\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.686980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686834    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-socket\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.686980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686874    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-certs\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.686980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686907    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-certs\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.686980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686912    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-data" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.686980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.686924    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-data" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.687430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.687013    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-credential-socket\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.687430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.687118    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-certs" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.687430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.687127    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-envoy\") pod \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\" (UID: \"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf\") "
Nov 03 19:12:14.687430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.687199    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istiod-ca-cert" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:14.687430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.687228    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-token\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-token\") pod \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\" (UID: \"4b63ba25-64d3-4ae9-8e7c-d7572376d097\") "
Nov 03 19:12:14.687694 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.687503    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-socket" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.687694 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.687550    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istiod-ca-cert" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:14.688031 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688008    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688182 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688038    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688182 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688054    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688195    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688213    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688227    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688242    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688244    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-certs" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.688328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688257    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.688653 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.688377    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-credential-socket" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.689724 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.689611    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-podinfo" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:12:14.690624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.689928    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-token" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:14.690624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.690400    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-podinfo" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:12:14.690743 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.690667    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-envoy" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.690800 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.690745    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-kube-api-access-9s2lt" pod "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" (UID: "cbe4fc5b-6e44-4231-8ebd-6b351867d0cf"). InnerVolumeSpecName "kube-api-access-9s2lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:14.690955 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.690935    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-envoy" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:14.691519 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.691491    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-token" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:14.691626 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.691608    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-kube-api-access-bbhdb" pod "4b63ba25-64d3-4ae9-8e7c-d7572376d097" (UID: "4b63ba25-64d3-4ae9-8e7c-d7572376d097"). InnerVolumeSpecName "kube-api-access-bbhdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:14.790199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790134    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790195    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790211    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790225    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790241    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2lt\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-kube-api-access-9s2lt\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790254    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790267    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790281    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhdb\" (UniqueName: \"kubernetes.io/projected/4b63ba25-64d3-4ae9-8e7c-d7572376d097-kube-api-access-bbhdb\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790295    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:14.790476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:14.790308    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b63ba25-64d3-4ae9-8e7c-d7572376d097-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:15.084369 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.084334    4489 generic.go:356] "Generic (PLEG): container finished" podID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerID="65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea" exitCode=0
Nov 03 19:12:15.084568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.084376    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"
Nov 03 19:12:15.084568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.084413    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" event={"ID":"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf","Type":"ContainerDied","Data":"65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea"}
Nov 03 19:12:15.084568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.084465    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm" event={"ID":"cbe4fc5b-6e44-4231-8ebd-6b351867d0cf","Type":"ContainerDied","Data":"f0704a92b1d8f047a4de974bec87c0ece343a74b4a6be8c6780410e0d862fc1e"}
Nov 03 19:12:15.084568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.084486    4489 scope.go:122] "RemoveContainer" containerID="65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea"
Nov 03 19:12:15.086510 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.086484    4489 generic.go:356] "Generic (PLEG): container finished" podID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerID="0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa" exitCode=0
Nov 03 19:12:15.086622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.086534    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" event={"ID":"4b63ba25-64d3-4ae9-8e7c-d7572376d097","Type":"ContainerDied","Data":"0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa"}
Nov 03 19:12:15.086622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.086564    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"
Nov 03 19:12:15.086622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.086598    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5" event={"ID":"4b63ba25-64d3-4ae9-8e7c-d7572376d097","Type":"ContainerDied","Data":"d2e305a11b668104d411a62781758e8a6855255a1b49c0b3dc37fb4f34a281c1"}
Nov 03 19:12:15.095543 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.095521    4489 scope.go:122] "RemoveContainer" containerID="65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea"
Nov 03 19:12:15.095864 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:12:15.095844    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea\": container with ID starting with 65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea not found: ID does not exist" containerID="65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea"
Nov 03 19:12:15.095992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.095879    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea"} err="failed to get container status \"65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea\": rpc error: code = NotFound desc = could not find container \"65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea\": container with ID starting with 65bc1642d5a3c305ced3183d6c99ad071ff3910fcdc133b67ea30d554ed634ea not found: ID does not exist"
Nov 03 19:12:15.095992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.095900    4489 scope.go:122] "RemoveContainer" containerID="0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa"
Nov 03 19:12:15.106382 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.106344    4489 scope.go:122] "RemoveContainer" containerID="0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa"
Nov 03 19:12:15.106690 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:12:15.106672    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa\": container with ID starting with 0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa not found: ID does not exist" containerID="0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa"
Nov 03 19:12:15.106728 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.106699    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa"} err="failed to get container status \"0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa\": rpc error: code = NotFound desc = could not find container \"0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa\": container with ID starting with 0b3c6831eb5b7aa1e1b8b86a482af7cdb3dd6fb0f30579063fce917fe8d15dfa not found: ID does not exist"
Nov 03 19:12:15.275572 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.275489    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"]
Nov 03 19:12:15.298257 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.298222    4489 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:12:15.298443 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.298292    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:12:15.308183 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.308117    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-c9kw5"]
Nov 03 19:12:15.391026 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.390992    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"]
Nov 03 19:12:15.394875 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:15.394849    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-vbkvm"]
Nov 03 19:12:16.298345 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:16.298308    4489 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:12:16.298511 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:16.298368    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:12:16.996679 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:16.996645    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" path="/var/lib/kubelet/pods/4b63ba25-64d3-4ae9-8e7c-d7572376d097/volumes"
Nov 03 19:12:16.997217 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:16.997182    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" path="/var/lib/kubelet/pods/cbe4fc5b-6e44-4231-8ebd-6b351867d0cf/volumes"
Nov 03 19:12:17.297777 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:17.297690    4489 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:12:17.297777 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:17.297749    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:12:17.534412 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:12:17.534373    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:12:18.297697 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:18.297655    4489 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:12:18.298317 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:18.297715    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:12:19.298205 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:19.298133    4489 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:12:19.298624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:19.298222    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:12:20.303319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:20.303287    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:20.303708 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:20.303590    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:20.304424 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:20.304407    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:12:21.255266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.255231    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"]
Nov 03 19:12:21.256208 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.256172    4489 cpu_manager.go:414] "containerMap: removing container" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy"
Nov 03 19:12:21.256208 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.256200    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy"
Nov 03 19:12:21.256406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.256252    4489 cpu_manager.go:414] "containerMap: removing container" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy"
Nov 03 19:12:21.256406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.256262    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy"
Nov 03 19:12:21.256406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.256387    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b63ba25-64d3-4ae9-8e7c-d7572376d097" containerName="istio-proxy"
Nov 03 19:12:21.256406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.256401    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe4fc5b-6e44-4231-8ebd-6b351867d0cf" containerName="istio-proxy"
Nov 03 19:12:21.260027 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.260003    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:21.313616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.313582    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"]
Nov 03 19:12:21.314490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.314463    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-69e2b9\""
Nov 03 19:12:21.371196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.371029    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqzq\" (UniqueName: \"kubernetes.io/projected/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-kube-api-access-jfqzq\") pod \"payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:21.374170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.371712    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:21.395294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.395259    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"]
Nov 03 19:12:21.399266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.399239    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.427199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.416975    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-69e2b9-dockercfg-zxq8q\""
Nov 03 19:12:21.458259 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.458224    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"]
Nov 03 19:12:21.491013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.490969    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.491543 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.491516    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:21.491741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.491722    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnpz\" (UniqueName: \"kubernetes.io/projected/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-kube-api-access-lbnpz\") pod \"payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.491933 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.491917    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqzq\" (UniqueName: \"kubernetes.io/projected/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-kube-api-access-jfqzq\") pod \"payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:21.493573 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.493526    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:21.596950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.596852    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.597435 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.597404    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnpz\" (UniqueName: \"kubernetes.io/projected/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-kube-api-access-lbnpz\") pod \"payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.599123 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.599044    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.612582 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.612352    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqzq\" (UniqueName: \"kubernetes.io/projected/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-kube-api-access-jfqzq\") pod \"payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:21.661043 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.661006    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnpz\" (UniqueName: \"kubernetes.io/projected/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-kube-api-access-lbnpz\") pod \"payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.749766 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.749728    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:21.874401 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:21.874312    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:22.125429 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:12:22.125364    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d52a40e_0e1d_48ef_91c2_bbf189492e1b.slice/crio-82a671e311e42406868c54ac0bf7175419f92e7bbd14124c6eddc1d31aecc6f9 WatchSource:0}: Error finding container 82a671e311e42406868c54ac0bf7175419f92e7bbd14124c6eddc1d31aecc6f9: Status 404 returned error can't find the container with id 82a671e311e42406868c54ac0bf7175419f92e7bbd14124c6eddc1d31aecc6f9
Nov 03 19:12:22.130528 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:22.130407    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"]
Nov 03 19:12:22.143954 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:22.143921    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw" event={"ID":"2d52a40e-0e1d-48ef-91c2-bbf189492e1b","Type":"ContainerStarted","Data":"82a671e311e42406868c54ac0bf7175419f92e7bbd14124c6eddc1d31aecc6f9"}
Nov 03 19:12:22.252226 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:12:22.252197    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6a6fb4_9ef1_422b_9d5e_e0a4da966e61.slice/crio-55309a8d7529aa178b17a0c3e49e5e5caa821dcebaeb01277338533e47ed3c9e WatchSource:0}: Error finding container 55309a8d7529aa178b17a0c3e49e5e5caa821dcebaeb01277338533e47ed3c9e: Status 404 returned error can't find the container with id 55309a8d7529aa178b17a0c3e49e5e5caa821dcebaeb01277338533e47ed3c9e
Nov 03 19:12:22.256656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:22.256633    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"]
Nov 03 19:12:23.151960 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:23.151921    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw" event={"ID":"2d52a40e-0e1d-48ef-91c2-bbf189492e1b","Type":"ContainerStarted","Data":"300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057"}
Nov 03 19:12:23.152491 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:23.152004    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:23.153699 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:23.153670    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq" event={"ID":"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61","Type":"ContainerStarted","Data":"cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4"}
Nov 03 19:12:23.153821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:23.153719    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq" event={"ID":"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61","Type":"ContainerStarted","Data":"55309a8d7529aa178b17a0c3e49e5e5caa821dcebaeb01277338533e47ed3c9e"}
Nov 03 19:12:23.153821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:23.153761    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:23.274034 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:23.273974    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw" podStartSLOduration=2.273954262 podStartE2EDuration="2.273954262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:23.271296399 +0000 UTC m=+2072.419990983" watchObservedRunningTime="2026-11-03 19:12:23.273954262 +0000 UTC m=+2072.422648834"
Nov 03 19:12:23.416264 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:23.416191    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq" podStartSLOduration=2.416172755 podStartE2EDuration="2.416172755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:23.405381199 +0000 UTC m=+2072.554075817" watchObservedRunningTime="2026-11-03 19:12:23.416172755 +0000 UTC m=+2072.564867327"
Nov 03 19:12:29.055470 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.055438    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"]
Nov 03 19:12:29.059716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.059695    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.102832 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.102791    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-metrics-certs\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.103033 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.103014    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.103089 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.103071    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2fbk\" (UniqueName: \"kubernetes.io/projected/44f00105-4d75-443d-bf04-5394b9953c34-kube-api-access-f2fbk\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.103293 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.103273    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f00105-4d75-443d-bf04-5394b9953c34-openshift-service-ca\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.115495 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.115461    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"]
Nov 03 19:12:29.163851 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.163819    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-efd81c-serving-cert\""
Nov 03 19:12:29.204508 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.204470    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.204695 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.204534    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2fbk\" (UniqueName: \"kubernetes.io/projected/44f00105-4d75-443d-bf04-5394b9953c34-kube-api-access-f2fbk\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.204695 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.204607    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f00105-4d75-443d-bf04-5394b9953c34-openshift-service-ca\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.204695 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:12:29.204624    4489 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-serving-cert: secret "maas-api-e2e-api-b-efd81c-serving-cert" not found
Nov 03 19:12:29.204695 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.204653    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-metrics-certs\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.204695 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:12:29.204688    4489 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls podName:44f00105-4d75-443d-bf04-5394b9953c34 nodeName:}" failed. No retries permitted until 2026-11-03 19:12:29.704668146 +0000 UTC m=+2078.853362711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls") pod "maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh" (UID: "44f00105-4d75-443d-bf04-5394b9953c34") : secret "maas-api-e2e-api-b-efd81c-serving-cert" not found
Nov 03 19:12:29.205388 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.205365    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f00105-4d75-443d-bf04-5394b9953c34-openshift-service-ca\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.207333 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.207313    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-metrics-certs\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.315190 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.314974    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2fbk\" (UniqueName: \"kubernetes.io/projected/44f00105-4d75-443d-bf04-5394b9953c34-kube-api-access-f2fbk\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.721669 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.721627    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.724384 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.724358    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls\") pod \"maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:29.973827 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:29.973738    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:30.352442 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:12:30.352402    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f00105_4d75_443d_bf04_5394b9953c34.slice/crio-4082ed7b843602876f8000ea9052d1179c99a247e639f4560c2ebd53668763d7 WatchSource:0}: Error finding container 4082ed7b843602876f8000ea9052d1179c99a247e639f4560c2ebd53668763d7: Status 404 returned error can't find the container with id 4082ed7b843602876f8000ea9052d1179c99a247e639f4560c2ebd53668763d7
Nov 03 19:12:30.354482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:30.354459    4489 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:12:30.358698 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:30.358673    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"]
Nov 03 19:12:31.212000 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:31.211966    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh" event={"ID":"44f00105-4d75-443d-bf04-5394b9953c34","Type":"ContainerStarted","Data":"4082ed7b843602876f8000ea9052d1179c99a247e639f4560c2ebd53668763d7"}
Nov 03 19:12:32.219514 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:32.219469    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh" event={"ID":"44f00105-4d75-443d-bf04-5394b9953c34","Type":"ContainerStarted","Data":"4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919"}
Nov 03 19:12:32.219955 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:32.219690    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:32.342190 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:32.342119    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh" podStartSLOduration=2.518621243 podStartE2EDuration="3.342105368s" totalImagesPullingTime="823.484125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:32.340630434 +0000 UTC m=+2081.489325006" watchObservedRunningTime="2026-11-03 19:12:32.342105368 +0000 UTC m=+2081.490799940"
Nov 03 19:12:34.164247 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:34.164213    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:12:34.165093 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:34.165068    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:12:38.231874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:12:38.231837    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:12:47.534405 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:12:47.534361    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:13:09.036719 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:09.036641    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"]
Nov 03 19:13:09.037273 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:09.036831    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" containerID="cri-o://a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431" gracePeriod=30
Nov 03 19:13:09.216878 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:09.216847    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"]
Nov 03 19:13:09.217102 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:09.217075    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" containerID="cri-o://2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702" gracePeriod=30
Nov 03 19:13:09.373094 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:09.372950    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"]
Nov 03 19:13:09.373318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:09.373242    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy" containerID="cri-o://6f708128a4ccf7863c5cf71a3447f45546d4347695810c4e335c3711ecd61227" gracePeriod=30
Nov 03 19:13:10.647118 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:10.647066    4489 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:13:10.647561 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:10.647137    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:13:12.656407 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:12.656365    4489 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:13:12.656780 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:12.656448    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:13:14.352881 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.352853    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 19:13:14.418159 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418110    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-podinfo\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418361 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418188    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-certs\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418361 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418237    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-envoy\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418361 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418286    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-socket\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418361 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418315    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-data\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418361 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418336    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-credential-socket\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418659 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418445    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-kube-api-access-rpqzt\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-kube-api-access-rpqzt\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418659 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418483    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-token\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-token\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.418659 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418512    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istiod-ca-cert\") pod \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\" (UID: \"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce\") "
Nov 03 19:13:14.419633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.418842    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-data" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.419633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.419168    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.419633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.419177    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istiod-ca-cert" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:14.419633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.419354    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-socket" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.419633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.419531    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-credential-socket" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.419633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.419556    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-certs" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.423039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.422990    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-envoy" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.423039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.422997    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-token" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:14.423194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.423044    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-podinfo" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:13:14.423194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.423116    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-kube-api-access-rpqzt" pod "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" (UID: "2dcaa9b1-b96c-49bd-8b46-968acf1c3cce"). InnerVolumeSpecName "kube-api-access-rpqzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:14.487911 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.487882    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:13:14.520423 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520395    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.520423 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520422    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.520612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520434    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.520612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520447    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.520612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520461    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.520612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520473    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpqzt\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-kube-api-access-rpqzt\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.520612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520487    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.520612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.520501    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.553209 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.553177    4489 generic.go:356] "Generic (PLEG): container finished" podID="1b67856f-b54a-4f38-a021-98e6d710f254" containerID="2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702" exitCode=0
Nov 03 19:13:14.553426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.553212    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" event={"ID":"1b67856f-b54a-4f38-a021-98e6d710f254","Type":"ContainerDied","Data":"2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702"}
Nov 03 19:13:14.553426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.553251    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"
Nov 03 19:13:14.553426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.553280    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk" event={"ID":"1b67856f-b54a-4f38-a021-98e6d710f254","Type":"ContainerDied","Data":"9d6d1b8ade1a86a09104d45a62a14ee489cc26415bd98d5edb5aa97d8cc0fed4"}
Nov 03 19:13:14.553426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.553312    4489 scope.go:122] "RemoveContainer" containerID="2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702"
Nov 03 19:13:14.555429 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.555361    4489 generic.go:356] "Generic (PLEG): container finished" podID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerID="6f708128a4ccf7863c5cf71a3447f45546d4347695810c4e335c3711ecd61227" exitCode=0
Nov 03 19:13:14.555692 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.555674    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" event={"ID":"5ded5c82-3bca-4fa0-b294-1864c3dff0bd","Type":"ContainerDied","Data":"6f708128a4ccf7863c5cf71a3447f45546d4347695810c4e335c3711ecd61227"}
Nov 03 19:13:14.558778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.558757    4489 generic.go:356] "Generic (PLEG): container finished" podID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerID="a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431" exitCode=0
Nov 03 19:13:14.558990 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.558813    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"
Nov 03 19:13:14.558990 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.558863    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" event={"ID":"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce","Type":"ContainerDied","Data":"a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431"}
Nov 03 19:13:14.558990 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.558904    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf" event={"ID":"2dcaa9b1-b96c-49bd-8b46-968acf1c3cce","Type":"ContainerDied","Data":"17bd66160078015720baba53197ab2ef3f989be6643883fa6ba7ee5216d2425b"}
Nov 03 19:13:14.567331 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.567308    4489 scope.go:122] "RemoveContainer" containerID="2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702"
Nov 03 19:13:14.567697 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:13:14.567664    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702\": container with ID starting with 2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702 not found: ID does not exist" containerID="2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702"
Nov 03 19:13:14.567793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.567708    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702"} err="failed to get container status \"2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702\": rpc error: code = NotFound desc = could not find container \"2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702\": container with ID starting with 2ba29e1efca3a523e60ed9abad2cef21b2c818e24c0b1df4f33e66f9e2010702 not found: ID does not exist"
Nov 03 19:13:14.567793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.567730    4489 scope.go:122] "RemoveContainer" containerID="a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431"
Nov 03 19:13:14.581676 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.581658    4489 scope.go:122] "RemoveContainer" containerID="a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431"
Nov 03 19:13:14.582035 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:13:14.582015    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431\": container with ID starting with a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431 not found: ID does not exist" containerID="a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431"
Nov 03 19:13:14.582106 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.582043    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431"} err="failed to get container status \"a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431\": rpc error: code = NotFound desc = could not find container \"a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431\": container with ID starting with a3b15c875f25d4f3666924d8a8ef996d7feed8e94b58206e1267e35f4ded3431 not found: ID does not exist"
Nov 03 19:13:14.621712 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.621671    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-data\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.621885 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.621756    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-envoy\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.621885 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.621827    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-kube-api-access-jwh9j\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-kube-api-access-jwh9j\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.622005 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.621884    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-socket\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.622005 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.621926    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b67856f-b54a-4f38-a021-98e6d710f254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b67856f-b54a-4f38-a021-98e6d710f254-istiod-ca-cert\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.622005 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.621967    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-istio-token\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-istio-token\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.622242 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.622051    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b67856f-b54a-4f38-a021-98e6d710f254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b67856f-b54a-4f38-a021-98e6d710f254-istio-podinfo\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.622242 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.622113    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-credential-socket\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.622242 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.622185    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-certs\") pod \"1b67856f-b54a-4f38-a021-98e6d710f254\" (UID: \"1b67856f-b54a-4f38-a021-98e6d710f254\") "
Nov 03 19:13:14.623461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.623418    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-socket" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.623573 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.623463    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-credential-socket" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.624299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.624256    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b67856f-b54a-4f38-a021-98e6d710f254-istiod-ca-cert" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:14.624485 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.624451    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-data" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.624809 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.624789    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-certs" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.626871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.626847    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-istio-token" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:14.627788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.627665    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-kube-api-access-jwh9j" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "kube-api-access-jwh9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:14.627788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.627759    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b67856f-b54a-4f38-a021-98e6d710f254-istio-podinfo" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:13:14.628492 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.628459    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-envoy" pod "1b67856f-b54a-4f38-a021-98e6d710f254" (UID: "1b67856f-b54a-4f38-a021-98e6d710f254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.671451 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.671421    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:13:14.724722 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724694    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724722 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724721    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwh9j\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-kube-api-access-jwh9j\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724733    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724743    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b67856f-b54a-4f38-a021-98e6d710f254-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724751    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b67856f-b54a-4f38-a021-98e6d710f254-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724761    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b67856f-b54a-4f38-a021-98e6d710f254-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724770    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724779    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.724918 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.724788    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b67856f-b54a-4f38-a021-98e6d710f254-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.826321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826235    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istiod-ca-cert\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.826321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826280    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-socket\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.826529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826382    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-kube-api-access-dmrm4\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-kube-api-access-dmrm4\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.826529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826434    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-certs\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.826529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826476    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-token\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-token\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.826529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826555    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-podinfo\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.826529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826683    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-socket" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.826529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826732    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istiod-ca-cert" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826789    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-certs" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826876    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-data\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.826957    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-credential-socket\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.827027    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-envoy\") pod \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\" (UID: \"5ded5c82-3bca-4fa0-b294-1864c3dff0bd\") "
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.827248    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-data" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.827349    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-credential-socket" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.828502    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.828539    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.828555    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.828568    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.828580    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.829567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.829093    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-kube-api-access-dmrm4" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "kube-api-access-dmrm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:14.829983 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.829626    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-podinfo" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:13:14.829983 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.829720    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-envoy" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:14.829983 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.829743    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-token" pod "5ded5c82-3bca-4fa0-b294-1864c3dff0bd" (UID: "5ded5c82-3bca-4fa0-b294-1864c3dff0bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:14.832284 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.832264    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"]
Nov 03 19:13:14.835355 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.835323    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-lh6wf"]
Nov 03 19:13:14.929686 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.929654    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.929686 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.929684    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.929686 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.929693    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.929913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.929703    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrm4\" (UniqueName: \"kubernetes.io/projected/5ded5c82-3bca-4fa0-b294-1864c3dff0bd-kube-api-access-dmrm4\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:14.998626 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:14.998595    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" path="/var/lib/kubelet/pods/2dcaa9b1-b96c-49bd-8b46-968acf1c3cce/volumes"
Nov 03 19:13:15.076616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:15.076550    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"]
Nov 03 19:13:15.081888 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:15.081852    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t4xjk"]
Nov 03 19:13:15.568208 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:15.568180    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"
Nov 03 19:13:15.568660 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:15.568182    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8" event={"ID":"5ded5c82-3bca-4fa0-b294-1864c3dff0bd","Type":"ContainerDied","Data":"27df0d8f3f1d4eb6a02f082e8fce6f5a8cf86db50b7364c965b2c5ba3183564b"}
Nov 03 19:13:15.568660 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:15.568325    4489 scope.go:122] "RemoveContainer" containerID="6f708128a4ccf7863c5cf71a3447f45546d4347695810c4e335c3711ecd61227"
Nov 03 19:13:15.746042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:15.746012    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"]
Nov 03 19:13:15.748924 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:15.748903    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-nkkk8"]
Nov 03 19:13:16.995961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:16.995927    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" path="/var/lib/kubelet/pods/1b67856f-b54a-4f38-a021-98e6d710f254/volumes"
Nov 03 19:13:16.996517 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:16.996500    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" path="/var/lib/kubelet/pods/5ded5c82-3bca-4fa0-b294-1864c3dff0bd/volumes"
Nov 03 19:13:17.531969 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:13:17.531937    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:13:28.684436 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.684404    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"]
Nov 03 19:13:28.684985 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.684971    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy"
Nov 03 19:13:28.685028 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.684985    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy"
Nov 03 19:13:28.685028 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.685010    4489 cpu_manager.go:414] "containerMap: removing container" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy"
Nov 03 19:13:28.685028 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.685016    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy"
Nov 03 19:13:28.685028 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.685022    4489 cpu_manager.go:414] "containerMap: removing container" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy"
Nov 03 19:13:28.685028 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.685027    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy"
Nov 03 19:13:28.685253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.685109    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcaa9b1-b96c-49bd-8b46-968acf1c3cce" containerName="istio-proxy"
Nov 03 19:13:28.685253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.685116    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b67856f-b54a-4f38-a021-98e6d710f254" containerName="istio-proxy"
Nov 03 19:13:28.685253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.685128    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ded5c82-3bca-4fa0-b294-1864c3dff0bd" containerName="istio-proxy"
Nov 03 19:13:28.690073 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.690057    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.743923 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.743888    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-certs\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744116 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744038    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istiod-ca-cert\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744116 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744111    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-socket\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744170    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-podinfo\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744211    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5df\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-kube-api-access-gt5df\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744257    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-credential-socket\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744425 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744285    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-envoy\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744425 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744372    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-data\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.744425 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.744400    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-token\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.757492 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.757464    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"]
Nov 03 19:13:28.796863 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.796834    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-65e151-openshift-default-dockercfg-gr7q9\""
Nov 03 19:13:28.871110 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.871070    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-certs\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.875165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.871461    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istiod-ca-cert\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.875492 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.875465    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-socket\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.875638 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.875623    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-podinfo\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.875790 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.875772    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5df\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-kube-api-access-gt5df\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.875937 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.875923    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-credential-socket\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.885264 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.885227    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-envoy\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.885566 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.885544    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-data\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.885717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.885701    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-token\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.886060 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.879216    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-credential-socket\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.886205 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.877924    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-certs\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.886611 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.886591    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-data\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.889983 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.878552    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istiod-ca-cert\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.890427 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.877624    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-socket\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.905743 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.905717    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-podinfo\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.930165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.916806    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-envoy\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.955993 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.955921    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-token\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:28.977177 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:28.967853    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5df\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-kube-api-access-gt5df\") pod \"e2e-inf-a-65e151-openshift-default-5478585457-cxqss\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") " pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:29.005253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.005219    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:29.434447 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:13:29.434410    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c251c64_ee71_4e62_99cc_40ca3e911ab1.slice/crio-bb3cab591fd2d151f4d9b822787f6c0a641158e5af365f2d5802a53e132544f3 WatchSource:0}: Error finding container bb3cab591fd2d151f4d9b822787f6c0a641158e5af365f2d5802a53e132544f3: Status 404 returned error can't find the container with id bb3cab591fd2d151f4d9b822787f6c0a641158e5af365f2d5802a53e132544f3
Nov 03 19:13:29.439324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.438307    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:13:29.439324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.438389    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:13:29.439324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.438435    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:13:29.439509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.439391    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"]
Nov 03 19:13:29.659262 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.659230    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" event={"ID":"8c251c64-ee71-4e62-99cc-40ca3e911ab1","Type":"ContainerStarted","Data":"693969bc77b65c4e37f37dc60510341943f1089a841b619bd4892dbb33b9760f"}
Nov 03 19:13:29.659448 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.659308    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" event={"ID":"8c251c64-ee71-4e62-99cc-40ca3e911ab1","Type":"ContainerStarted","Data":"bb3cab591fd2d151f4d9b822787f6c0a641158e5af365f2d5802a53e132544f3"}
Nov 03 19:13:29.796220 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:29.794515    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podStartSLOduration=1.794496372 podStartE2EDuration="1.794496372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:29.792655194 +0000 UTC m=+2138.941349766" watchObservedRunningTime="2026-11-03 19:13:29.794496372 +0000 UTC m=+2138.943190944"
Nov 03 19:13:30.006173 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:30.006103    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:30.007820 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:30.007790    4489 patch_prober.go:29] interesting pod/e2e-inf-a-65e151-openshift-default-5478585457-cxqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:13:30.007970 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:30.007878    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:13:31.010490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:31.010465    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:31.670874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:31.670842    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:31.671696 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:31.671674    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:13:36.255567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.255527    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"]
Nov 03 19:13:36.259863 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.259843    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:36.315925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.315892    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"]
Nov 03 19:13:36.369166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.367589    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-65e151-dockercfg-xh6gw\""
Nov 03 19:13:36.369166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.367693    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-65e151\""
Nov 03 19:13:36.431168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.430545    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"]
Nov 03 19:13:36.442297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.441536    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgqps\" (UniqueName: \"kubernetes.io/projected/65eb0dc6-a873-437e-a958-8a48edbc2158-kube-api-access-vgqps\") pod \"payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:36.442297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.441677    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eb0dc6-a873-437e-a958-8a48edbc2158-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:36.488277 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.488243    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"]
Nov 03 19:13:36.488466 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.488368    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.554168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.549094    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgqps\" (UniqueName: \"kubernetes.io/projected/65eb0dc6-a873-437e-a958-8a48edbc2158-kube-api-access-vgqps\") pod \"payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:36.554168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.549254    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eb0dc6-a873-437e-a958-8a48edbc2158-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:36.569648 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.560465    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eb0dc6-a873-437e-a958-8a48edbc2158-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:36.627849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.627814    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgqps\" (UniqueName: \"kubernetes.io/projected/65eb0dc6-a873-437e-a958-8a48edbc2158-kube-api-access-vgqps\") pod \"payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:36.651180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.651156    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4j4\" (UniqueName: \"kubernetes.io/projected/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-kube-api-access-gr4j4\") pod \"payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.651319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.651207    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.752914 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.752857    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4j4\" (UniqueName: \"kubernetes.io/projected/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-kube-api-access-gr4j4\") pod \"payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.753123 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.752956    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.753803 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.753777    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.816602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.816538    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4j4\" (UniqueName: \"kubernetes.io/projected/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-kube-api-access-gr4j4\") pod \"payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.826101 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.826067    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:36.877163 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:36.876491    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:37.200280 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:13:37.200252    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ddfc62_ffe9_4e3b_aeee_44aea01eef58.slice/crio-5e04a79fa29b6c9c61b6ee6d653098616c3fd8daf15ebb8b154b96a16184c330 WatchSource:0}: Error finding container 5e04a79fa29b6c9c61b6ee6d653098616c3fd8daf15ebb8b154b96a16184c330: Status 404 returned error can't find the container with id 5e04a79fa29b6c9c61b6ee6d653098616c3fd8daf15ebb8b154b96a16184c330
Nov 03 19:13:37.204458 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.204434    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"]
Nov 03 19:13:37.294561 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:13:37.294514    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65eb0dc6_a873_437e_a958_8a48edbc2158.slice/crio-53ce5f8adda576ce0bc26d4515b2ba48ecd467b8a2535f9d56d7ad9e5b5c842b WatchSource:0}: Error finding container 53ce5f8adda576ce0bc26d4515b2ba48ecd467b8a2535f9d56d7ad9e5b5c842b: Status 404 returned error can't find the container with id 53ce5f8adda576ce0bc26d4515b2ba48ecd467b8a2535f9d56d7ad9e5b5c842b
Nov 03 19:13:37.298978 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.298956    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"]
Nov 03 19:13:37.713381 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.713338    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27" event={"ID":"65eb0dc6-a873-437e-a958-8a48edbc2158","Type":"ContainerStarted","Data":"83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e"}
Nov 03 19:13:37.713568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.713408    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:37.713568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.713430    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27" event={"ID":"65eb0dc6-a873-437e-a958-8a48edbc2158","Type":"ContainerStarted","Data":"53ce5f8adda576ce0bc26d4515b2ba48ecd467b8a2535f9d56d7ad9e5b5c842b"}
Nov 03 19:13:37.715021 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.714992    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd" event={"ID":"97ddfc62-ffe9-4e3b-aeee-44aea01eef58","Type":"ContainerStarted","Data":"cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097"}
Nov 03 19:13:37.715154 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.715036    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd" event={"ID":"97ddfc62-ffe9-4e3b-aeee-44aea01eef58","Type":"ContainerStarted","Data":"5e04a79fa29b6c9c61b6ee6d653098616c3fd8daf15ebb8b154b96a16184c330"}
Nov 03 19:13:37.715154 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.715057    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:13:37.837065 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:37.837020    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27" podStartSLOduration=1.837006523 podStartE2EDuration="1.837006523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:37.833843538 +0000 UTC m=+2146.982538108" watchObservedRunningTime="2026-11-03 19:13:37.837006523 +0000 UTC m=+2146.985701094"
Nov 03 19:13:39.456187 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:39.453888    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd" podStartSLOduration=3.453857309 podStartE2EDuration="3.453857309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:37.957396724 +0000 UTC m=+2147.106091323" watchObservedRunningTime="2026-11-03 19:13:39.453857309 +0000 UTC m=+2148.602551890"
Nov 03 19:13:39.456187 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:39.454626    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"]
Nov 03 19:13:39.456187 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:39.454890    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" containerID="cri-o://45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2" gracePeriod=30
Nov 03 19:13:40.661109 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:40.661073    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:13:40.661484 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:40.661170    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:13:44.758971 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.758903    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 19:13:44.764300 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.764272    4489 generic.go:356] "Generic (PLEG): container finished" podID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerID="45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2" exitCode=0
Nov 03 19:13:44.764456 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.764311    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"
Nov 03 19:13:44.764456 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.764346    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" event={"ID":"05d98c68-36da-4e9a-8f48-79e378d0729c","Type":"ContainerDied","Data":"45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2"}
Nov 03 19:13:44.764456 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.764403    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9" event={"ID":"05d98c68-36da-4e9a-8f48-79e378d0729c","Type":"ContainerDied","Data":"670e9275a061280d424f8830c2060863bf07318403e6e5e7b9f7adad083a9325"}
Nov 03 19:13:44.764456 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.764433    4489 scope.go:122] "RemoveContainer" containerID="45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2"
Nov 03 19:13:44.776716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.776696    4489 scope.go:122] "RemoveContainer" containerID="45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2"
Nov 03 19:13:44.777039 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:13:44.777001    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2\": container with ID starting with 45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2 not found: ID does not exist" containerID="45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2"
Nov 03 19:13:44.777135 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.777046    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2"} err="failed to get container status \"45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2\": rpc error: code = NotFound desc = could not find container \"45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2\": container with ID starting with 45caddcba1bc9621e363931c1c7dbc7f83ea6cea50fdd5e3dbe272ec13de5da2 not found: ID does not exist"
Nov 03 19:13:44.859621 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859581    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-socket\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.859964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859646    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-token\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-token\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.859964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859675    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-credential-socket\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.859964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859742    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-data\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.859964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859770    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-certs\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.859964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859889    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-envoy\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.859964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859935    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-socket" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:44.859964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.859952    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-kube-api-access-vw752\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-kube-api-access-vw752\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.860461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.860019    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d98c68-36da-4e9a-8f48-79e378d0729c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d98c68-36da-4e9a-8f48-79e378d0729c-istiod-ca-cert\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.860461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.860166    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-data" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:44.860461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.860192    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-podinfo\") pod \"05d98c68-36da-4e9a-8f48-79e378d0729c\" (UID: \"05d98c68-36da-4e9a-8f48-79e378d0729c\") "
Nov 03 19:13:44.860774 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.860752    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.860774 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.860771    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.860972 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.860954    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-credential-socket" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:44.861303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.861275    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-certs" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:44.861483 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.861458    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d98c68-36da-4e9a-8f48-79e378d0729c-istiod-ca-cert" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:44.862638 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.862603    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-podinfo" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:13:44.863121 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.863099    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-envoy" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:44.863222 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.863114    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-token" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:44.863435 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.863417    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-kube-api-access-vw752" pod "05d98c68-36da-4e9a-8f48-79e378d0729c" (UID: "05d98c68-36da-4e9a-8f48-79e378d0729c"). InnerVolumeSpecName "kube-api-access-vw752". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:44.961998 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.961968    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.961998 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.961996    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw752\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-kube-api-access-vw752\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.962188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.962006    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d98c68-36da-4e9a-8f48-79e378d0729c-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.962188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.962015    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.962188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.962023    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d98c68-36da-4e9a-8f48-79e378d0729c-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.962188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.962034    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:44.962188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:44.962041    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d98c68-36da-4e9a-8f48-79e378d0729c-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:45.186906 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:45.186719    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"]
Nov 03 19:13:45.192154 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:45.192101    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-8vrc9"]
Nov 03 19:13:46.995841 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:46.995802    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" path="/var/lib/kubelet/pods/05d98c68-36da-4e9a-8f48-79e378d0729c/volumes"
Nov 03 19:13:47.532598 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:13:47.532562    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:13:48.727621 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:48.727589    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:13:48.728429 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:13:48.728404    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:14:17.533915 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:14:17.533880    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:14:35.748769 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:35.748692    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"]
Nov 03 19:14:40.749234 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:40.749185    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4" podUID="1769d8b3-2246-4dca-8bb4-65f32f77f192" containerName="payload-processing" containerID="cri-o://36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15" gracePeriod=25
Nov 03 19:14:40.762935 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:14:40.762913    4489 logging.go:55] [core] [Channel #1298 SubChannel #1299]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.89:50052", ServerName: "10.133.0.89:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.89:50052: connect: connection refused"
Nov 03 19:14:41.032085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.032058    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:14:41.050975 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.050941    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1769d8b3-2246-4dca-8bb4-65f32f77f192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769d8b3-2246-4dca-8bb4-65f32f77f192-plugins-config-volume\") pod \"1769d8b3-2246-4dca-8bb4-65f32f77f192\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") "
Nov 03 19:14:41.051104 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.051077    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1769d8b3-2246-4dca-8bb4-65f32f77f192-kube-api-access-cwrrl\" (UniqueName: \"kubernetes.io/projected/1769d8b3-2246-4dca-8bb4-65f32f77f192-kube-api-access-cwrrl\") pod \"1769d8b3-2246-4dca-8bb4-65f32f77f192\" (UID: \"1769d8b3-2246-4dca-8bb4-65f32f77f192\") "
Nov 03 19:14:41.051370 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.051347    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1769d8b3-2246-4dca-8bb4-65f32f77f192-plugins-config-volume" pod "1769d8b3-2246-4dca-8bb4-65f32f77f192" (UID: "1769d8b3-2246-4dca-8bb4-65f32f77f192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:14:41.051971 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.051949    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1769d8b3-2246-4dca-8bb4-65f32f77f192-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:41.053328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.053305    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1769d8b3-2246-4dca-8bb4-65f32f77f192-kube-api-access-cwrrl" pod "1769d8b3-2246-4dca-8bb4-65f32f77f192" (UID: "1769d8b3-2246-4dca-8bb4-65f32f77f192"). InnerVolumeSpecName "kube-api-access-cwrrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:14:41.139815 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.139788    4489 generic.go:356] "Generic (PLEG): container finished" podID="1769d8b3-2246-4dca-8bb4-65f32f77f192" containerID="36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15" exitCode=0
Nov 03 19:14:41.139958 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.139904    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"
Nov 03 19:14:41.139958 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.139921    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4" event={"ID":"1769d8b3-2246-4dca-8bb4-65f32f77f192","Type":"ContainerDied","Data":"36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15"}
Nov 03 19:14:41.140027 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.139973    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4" event={"ID":"1769d8b3-2246-4dca-8bb4-65f32f77f192","Type":"ContainerDied","Data":"2989a1a0f1ce4f5d071f7e10d6217eac986c7c10000709f35dd412e3581a1a65"}
Nov 03 19:14:41.140027 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.140002    4489 scope.go:122] "RemoveContainer" containerID="36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15"
Nov 03 19:14:41.151125 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.151102    4489 scope.go:122] "RemoveContainer" containerID="36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15"
Nov 03 19:14:41.151487 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:14:41.151456    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15\": container with ID starting with 36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15 not found: ID does not exist" containerID="36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15"
Nov 03 19:14:41.151583 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.151495    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15"} err="failed to get container status \"36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15\": rpc error: code = NotFound desc = could not find container \"36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15\": container with ID starting with 36de6f54839e47ed42eb16f771e8f226333941dcc6a2ce55039a73269f161f15 not found: ID does not exist"
Nov 03 19:14:41.153067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.153046    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrrl\" (UniqueName: \"kubernetes.io/projected/1769d8b3-2246-4dca-8bb4-65f32f77f192-kube-api-access-cwrrl\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:41.270776 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.270699    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"]
Nov 03 19:14:41.274813 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.274787    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4"]
Nov 03 19:14:41.763372 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:41.763331    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4320e5-5854dbd4c-bkzr4" podUID="1769d8b3-2246-4dca-8bb4-65f32f77f192" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.89:50052\" within 1s: context deadline exceeded"
Nov 03 19:14:42.995593 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:42.995553    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1769d8b3-2246-4dca-8bb4-65f32f77f192" path="/var/lib/kubelet/pods/1769d8b3-2246-4dca-8bb4-65f32f77f192/volumes"
Nov 03 19:14:47.549565 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:14:47.549531    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:14:58.608394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:58.608362    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"]
Nov 03 19:14:58.608893 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:58.608651    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" containerID="cri-o://89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834" gracePeriod=30
Nov 03 19:14:58.829862 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:58.829821    4489 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:14:58.830047 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:14:58.829911    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:15:00.204642 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.204603    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"]
Nov 03 19:15:00.205246 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.205231    4489 cpu_manager.go:414] "containerMap: removing container" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy"
Nov 03 19:15:00.205303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.205248    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy"
Nov 03 19:15:00.205303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.205258    4489 cpu_manager.go:414] "containerMap: removing container" podUID="1769d8b3-2246-4dca-8bb4-65f32f77f192" containerName="payload-processing"
Nov 03 19:15:00.205303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.205263    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1769d8b3-2246-4dca-8bb4-65f32f77f192" containerName="payload-processing"
Nov 03 19:15:00.205399 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.205362    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d98c68-36da-4e9a-8f48-79e378d0729c" containerName="istio-proxy"
Nov 03 19:15:00.205399 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.205370    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="1769d8b3-2246-4dca-8bb4-65f32f77f192" containerName="payload-processing"
Nov 03 19:15:00.210045 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.210025    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.229891 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.229866    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"]
Nov 03 19:15:00.233611 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.233595    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.237452 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.237432    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"]
Nov 03 19:15:00.241376 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.241358    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.259035 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.259013    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"]
Nov 03 19:15:00.264076 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.264058    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.278713 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.278692    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\""
Nov 03 19:15:00.316358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.316331    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"]
Nov 03 19:15:00.339645 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.339609    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzxf\" (UniqueName: \"kubernetes.io/projected/93065652-85d3-4d61-94dd-af4cdabc7651-kube-api-access-qfzxf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.339811 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.339660    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b14985-244a-4ca7-b977-38cb452421e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.339874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.339829    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4m95\" (UniqueName: \"kubernetes.io/projected/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-kube-api-access-k4m95\") pod \"maas-api-key-cleanup-29895555-hmrwr\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.339929 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.339890    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-hmrwr\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.340036 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.340013    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr54b\" (UniqueName: \"kubernetes.io/projected/4c1d06d1-840f-4e44-8740-f42b804dae84-kube-api-access-xr54b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.340133 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.340114    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d06d1-840f-4e44-8740-f42b804dae84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.340242 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.340225    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93065652-85d3-4d61-94dd-af4cdabc7651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.340350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.340328    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtwtk\" (UniqueName: \"kubernetes.io/projected/02b14985-244a-4ca7-b977-38cb452421e3-kube-api-access-mtwtk\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.367481 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.367449    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"]
Nov 03 19:15:00.373073 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.373045    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"]
Nov 03 19:15:00.411845 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.411818    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"]
Nov 03 19:15:00.442223 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442168    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d06d1-840f-4e44-8740-f42b804dae84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.442406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442284    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93065652-85d3-4d61-94dd-af4cdabc7651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.442406 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442365    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwtk\" (UniqueName: \"kubernetes.io/projected/02b14985-244a-4ca7-b977-38cb452421e3-kube-api-access-mtwtk\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.442548 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442486    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzxf\" (UniqueName: \"kubernetes.io/projected/93065652-85d3-4d61-94dd-af4cdabc7651-kube-api-access-qfzxf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.442548 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442518    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b14985-244a-4ca7-b977-38cb452421e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.442702 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442604    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4m95\" (UniqueName: \"kubernetes.io/projected/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-kube-api-access-k4m95\") pod \"maas-api-key-cleanup-29895555-hmrwr\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.442702 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442653    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-hmrwr\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.442839 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442699    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr54b\" (UniqueName: \"kubernetes.io/projected/4c1d06d1-840f-4e44-8740-f42b804dae84-kube-api-access-xr54b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.442943 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.442921    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d06d1-840f-4e44-8740-f42b804dae84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.443817 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.443790    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b14985-244a-4ca7-b977-38cb452421e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.443922 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.443790    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93065652-85d3-4d61-94dd-af4cdabc7651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.444430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.444408    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895555-hmrwr\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.509602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.509527    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr54b\" (UniqueName: \"kubernetes.io/projected/4c1d06d1-840f-4e44-8740-f42b804dae84-kube-api-access-xr54b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.509602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.509527    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4m95\" (UniqueName: \"kubernetes.io/projected/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-kube-api-access-k4m95\") pod \"maas-api-key-cleanup-29895555-hmrwr\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.510432 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.510409    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwtk\" (UniqueName: \"kubernetes.io/projected/02b14985-244a-4ca7-b977-38cb452421e3-kube-api-access-mtwtk\") pod \"maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.512815 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.512788    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzxf\" (UniqueName: \"kubernetes.io/projected/93065652-85d3-4d61-94dd-af4cdabc7651-kube-api-access-qfzxf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.524013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.523988    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:00.555204 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.555172    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:00.563944 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.563926    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:00.576411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.576037    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:00.904905 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:15:00.904870    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cbb9d7_bcb3_4b4d_be31_e0640c507f6c.slice/crio-b52383244d4c014fe71593f6a3826eee5e650115ff0993527aa054ef3753db37 WatchSource:0}: Error finding container b52383244d4c014fe71593f6a3826eee5e650115ff0993527aa054ef3753db37: Status 404 returned error can't find the container with id b52383244d4c014fe71593f6a3826eee5e650115ff0993527aa054ef3753db37
Nov 03 19:15:00.909878 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.909845    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"]
Nov 03 19:15:00.952846 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:15:00.952818    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b14985_244a_4ca7_b977_38cb452421e3.slice/crio-709b9b3016897cec8b9213c6337c7ecd2f6ff3f20ca81cca6c159dfe5d8864fa WatchSource:0}: Error finding container 709b9b3016897cec8b9213c6337c7ecd2f6ff3f20ca81cca6c159dfe5d8864fa: Status 404 returned error can't find the container with id 709b9b3016897cec8b9213c6337c7ecd2f6ff3f20ca81cca6c159dfe5d8864fa
Nov 03 19:15:00.957019 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.956998    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"]
Nov 03 19:15:00.999242 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:00.999216    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"]
Nov 03 19:15:01.002482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:01.002459    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"]
Nov 03 19:15:01.051961 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:15:01.051931    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93065652_85d3_4d61_94dd_af4cdabc7651.slice/crio-ac1133bdc59ee25c17e6bd032edd870dfa82554c21e8e466734bfe8173b4887c WatchSource:0}: Error finding container ac1133bdc59ee25c17e6bd032edd870dfa82554c21e8e466734bfe8173b4887c: Status 404 returned error can't find the container with id ac1133bdc59ee25c17e6bd032edd870dfa82554c21e8e466734bfe8173b4887c
Nov 03 19:15:01.052791 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:15:01.052768    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1d06d1_840f_4e44_8740_f42b804dae84.slice/crio-1602b746af5264dc3855f350d1009d2b5d30312ae2521f827eedb0cb880e507d WatchSource:0}: Error finding container 1602b746af5264dc3855f350d1009d2b5d30312ae2521f827eedb0cb880e507d: Status 404 returned error can't find the container with id 1602b746af5264dc3855f350d1009d2b5d30312ae2521f827eedb0cb880e507d
Nov 03 19:15:01.293169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:01.293125    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" event={"ID":"02b14985-244a-4ca7-b977-38cb452421e3","Type":"ContainerStarted","Data":"709b9b3016897cec8b9213c6337c7ecd2f6ff3f20ca81cca6c159dfe5d8864fa"}
Nov 03 19:15:01.294725 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:01.294702    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" event={"ID":"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c","Type":"ContainerStarted","Data":"dd1f39a5966c8b19df6b8f48059e143026b35bc8b915a3a5d6a793c2dc2d76c2"}
Nov 03 19:15:01.294841 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:01.294750    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" event={"ID":"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c","Type":"ContainerStarted","Data":"b52383244d4c014fe71593f6a3826eee5e650115ff0993527aa054ef3753db37"}
Nov 03 19:15:01.295981 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:01.295956    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" event={"ID":"93065652-85d3-4d61-94dd-af4cdabc7651","Type":"ContainerStarted","Data":"ac1133bdc59ee25c17e6bd032edd870dfa82554c21e8e466734bfe8173b4887c"}
Nov 03 19:15:01.297304 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:01.297271    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" event={"ID":"4c1d06d1-840f-4e44-8740-f42b804dae84","Type":"ContainerStarted","Data":"1602b746af5264dc3855f350d1009d2b5d30312ae2521f827eedb0cb880e507d"}
Nov 03 19:15:01.415072 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:01.414973    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" podStartSLOduration=1.41495808 podStartE2EDuration="1.41495808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:01.412762348 +0000 UTC m=+2230.561456920" watchObservedRunningTime="2026-11-03 19:15:01.41495808 +0000 UTC m=+2230.563652651"
Nov 03 19:15:02.311672 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.311639    4489 generic.go:356] "Generic (PLEG): container finished" podID="02b14985-244a-4ca7-b977-38cb452421e3" containerID="90769c9785285503b309fa7c5fd857147461dd682b8520ce83a56ff53b77e16e" exitCode=60
Nov 03 19:15:02.312104 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.311724    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" event={"ID":"02b14985-244a-4ca7-b977-38cb452421e3","Type":"ContainerDied","Data":"90769c9785285503b309fa7c5fd857147461dd682b8520ce83a56ff53b77e16e"}
Nov 03 19:15:02.312104 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.312018    4489 scope.go:122] "RemoveContainer" containerID="90769c9785285503b309fa7c5fd857147461dd682b8520ce83a56ff53b77e16e"
Nov 03 19:15:02.313372 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.313339    4489 generic.go:356] "Generic (PLEG): container finished" podID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerID="dd1f39a5966c8b19df6b8f48059e143026b35bc8b915a3a5d6a793c2dc2d76c2" exitCode=60
Nov 03 19:15:02.313499 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.313420    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" event={"ID":"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c","Type":"ContainerDied","Data":"dd1f39a5966c8b19df6b8f48059e143026b35bc8b915a3a5d6a793c2dc2d76c2"}
Nov 03 19:15:02.313650 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.313633    4489 scope.go:122] "RemoveContainer" containerID="dd1f39a5966c8b19df6b8f48059e143026b35bc8b915a3a5d6a793c2dc2d76c2"
Nov 03 19:15:02.315221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.315119    4489 generic.go:356] "Generic (PLEG): container finished" podID="93065652-85d3-4d61-94dd-af4cdabc7651" containerID="f2ddbe025123b41940896e20737bc9e2ba4b5f7675f75879cf9b63118ee07482" exitCode=60
Nov 03 19:15:02.315301 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.315215    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" event={"ID":"93065652-85d3-4d61-94dd-af4cdabc7651","Type":"ContainerDied","Data":"f2ddbe025123b41940896e20737bc9e2ba4b5f7675f75879cf9b63118ee07482"}
Nov 03 19:15:02.315463 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.315444    4489 scope.go:122] "RemoveContainer" containerID="f2ddbe025123b41940896e20737bc9e2ba4b5f7675f75879cf9b63118ee07482"
Nov 03 19:15:02.317085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.317063    4489 generic.go:356] "Generic (PLEG): container finished" podID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerID="2be96aa3e85e5839bd303ac49757f249963bc29d4c4c10375e4f2529fe8f5306" exitCode=60
Nov 03 19:15:02.317203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.317181    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" event={"ID":"4c1d06d1-840f-4e44-8740-f42b804dae84","Type":"ContainerDied","Data":"2be96aa3e85e5839bd303ac49757f249963bc29d4c4c10375e4f2529fe8f5306"}
Nov 03 19:15:02.317353 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:02.317338    4489 scope.go:122] "RemoveContainer" containerID="2be96aa3e85e5839bd303ac49757f249963bc29d4c4c10375e4f2529fe8f5306"
Nov 03 19:15:03.325202 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.325126    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" event={"ID":"02b14985-244a-4ca7-b977-38cb452421e3","Type":"ContainerStarted","Data":"5ac9c630fc9cf41ef198ab4ce61557e76008dda9663cc0d726cef8cecafa895f"}
Nov 03 19:15:03.327078 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.327052    4489 generic.go:356] "Generic (PLEG): container finished" podID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerID="21b198b921264a15c00881b688153368f0918963d1b7c2c020b6085cddab5bf8" exitCode=60
Nov 03 19:15:03.327191 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.327132    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" event={"ID":"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c","Type":"ContainerDied","Data":"21b198b921264a15c00881b688153368f0918963d1b7c2c020b6085cddab5bf8"}
Nov 03 19:15:03.327274 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.327206    4489 scope.go:122] "RemoveContainer" containerID="dd1f39a5966c8b19df6b8f48059e143026b35bc8b915a3a5d6a793c2dc2d76c2"
Nov 03 19:15:03.327496 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.327477    4489 scope.go:122] "RemoveContainer" containerID="21b198b921264a15c00881b688153368f0918963d1b7c2c020b6085cddab5bf8"
Nov 03 19:15:03.327864 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:03.327835    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-hmrwr_odh-ai-gateway-infra(f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c"
Nov 03 19:15:03.329199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.329175    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" event={"ID":"93065652-85d3-4d61-94dd-af4cdabc7651","Type":"ContainerStarted","Data":"413de92d9609680b68d426e74b4ee53b82a2c93eae017ea1afc604b92dfe5d8b"}
Nov 03 19:15:03.331040 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.331018    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" event={"ID":"4c1d06d1-840f-4e44-8740-f42b804dae84","Type":"ContainerStarted","Data":"d047179dceb071b7e8c4831406c9fd3dfae0c7234ddea55d4c3b7891339c2ad0"}
Nov 03 19:15:03.448269 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.448213    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" podStartSLOduration=3.448195865 podStartE2EDuration="3.448195865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:03.446351131 +0000 UTC m=+2232.595045702" watchObservedRunningTime="2026-11-03 19:15:03.448195865 +0000 UTC m=+2232.596890436"
Nov 03 19:15:03.604007 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.603944    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" podStartSLOduration=3.603924931 podStartE2EDuration="3.603924931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:03.603522495 +0000 UTC m=+2232.752217048" watchObservedRunningTime="2026-11-03 19:15:03.603924931 +0000 UTC m=+2232.752619503"
Nov 03 19:15:03.730692 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.730644    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" podStartSLOduration=3.730630048 podStartE2EDuration="3.730630048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:03.728269802 +0000 UTC m=+2232.876964374" watchObservedRunningTime="2026-11-03 19:15:03.730630048 +0000 UTC m=+2232.879324651"
Nov 03 19:15:03.903763 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:03.903739    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:15:04.007650 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007563    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319b6ca2-9d70-4ac4-b521-c5868508fb51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319b6ca2-9d70-4ac4-b521-c5868508fb51-istiod-ca-cert\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.007794 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007662    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-kube-api-access-zvkpw\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-kube-api-access-zvkpw\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.007794 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007708    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-socket\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.007794 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007763    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-podinfo\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.007920 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007807    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-credential-socket\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.007920 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007834    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-data\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.007920 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007907    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-envoy\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.008063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007948    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-token\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-token\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.008063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.007946    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319b6ca2-9d70-4ac4-b521-c5868508fb51-istiod-ca-cert" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:04.008063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008022    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-certs\") pod \"319b6ca2-9d70-4ac4-b521-c5868508fb51\" (UID: \"319b6ca2-9d70-4ac4-b521-c5868508fb51\") "
Nov 03 19:15:04.008256 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008082    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-socket" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:04.008256 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008100    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-credential-socket" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:04.008391 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008367    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-certs" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:04.008495 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008470    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-data" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:04.008648 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008633    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.008717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008656    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.008717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008669    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.008717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008683    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319b6ca2-9d70-4ac4-b521-c5868508fb51-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.008717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.008697    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.010766 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.010735    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-kube-api-access-zvkpw" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "kube-api-access-zvkpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:04.010854 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.010780    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-podinfo" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:15:04.010854 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.010828    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-token" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:04.011019 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.010993    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-envoy" pod "319b6ca2-9d70-4ac4-b521-c5868508fb51" (UID: "319b6ca2-9d70-4ac4-b521-c5868508fb51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:04.110904 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.110873    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvkpw\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-kube-api-access-zvkpw\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.110904 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.110901    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.110904 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.110912    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.111123 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.110922    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319b6ca2-9d70-4ac4-b521-c5868508fb51-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:04.341910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.341821    4489 scope.go:122] "RemoveContainer" containerID="21b198b921264a15c00881b688153368f0918963d1b7c2c020b6085cddab5bf8"
Nov 03 19:15:04.342777 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:04.342741    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-hmrwr_odh-ai-gateway-infra(f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c"
Nov 03 19:15:04.343597 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.343572    4489 generic.go:356] "Generic (PLEG): container finished" podID="93065652-85d3-4d61-94dd-af4cdabc7651" containerID="413de92d9609680b68d426e74b4ee53b82a2c93eae017ea1afc604b92dfe5d8b" exitCode=60
Nov 03 19:15:04.343702 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.343655    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" event={"ID":"93065652-85d3-4d61-94dd-af4cdabc7651","Type":"ContainerDied","Data":"413de92d9609680b68d426e74b4ee53b82a2c93eae017ea1afc604b92dfe5d8b"}
Nov 03 19:15:04.343746 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.343721    4489 scope.go:122] "RemoveContainer" containerID="f2ddbe025123b41940896e20737bc9e2ba4b5f7675f75879cf9b63118ee07482"
Nov 03 19:15:04.344084 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.344065    4489 scope.go:122] "RemoveContainer" containerID="413de92d9609680b68d426e74b4ee53b82a2c93eae017ea1afc604b92dfe5d8b"
Nov 03 19:15:04.344411 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:04.344387    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz_odh-ai-gateway-infra(93065652-85d3-4d61-94dd-af4cdabc7651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" podUID="93065652-85d3-4d61-94dd-af4cdabc7651"
Nov 03 19:15:04.345686 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.345665    4489 generic.go:356] "Generic (PLEG): container finished" podID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerID="d047179dceb071b7e8c4831406c9fd3dfae0c7234ddea55d4c3b7891339c2ad0" exitCode=60
Nov 03 19:15:04.345782 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.345750    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" event={"ID":"4c1d06d1-840f-4e44-8740-f42b804dae84","Type":"ContainerDied","Data":"d047179dceb071b7e8c4831406c9fd3dfae0c7234ddea55d4c3b7891339c2ad0"}
Nov 03 19:15:04.345978 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.345962    4489 scope.go:122] "RemoveContainer" containerID="d047179dceb071b7e8c4831406c9fd3dfae0c7234ddea55d4c3b7891339c2ad0"
Nov 03 19:15:04.346255 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:04.346234    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr_odh-ai-gateway-infra(4c1d06d1-840f-4e44-8740-f42b804dae84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84"
Nov 03 19:15:04.347598 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.347575    4489 generic.go:356] "Generic (PLEG): container finished" podID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerID="89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834" exitCode=0
Nov 03 19:15:04.347679 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.347668    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"
Nov 03 19:15:04.347719 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.347705    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" event={"ID":"319b6ca2-9d70-4ac4-b521-c5868508fb51","Type":"ContainerDied","Data":"89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834"}
Nov 03 19:15:04.347757 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.347748    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7" event={"ID":"319b6ca2-9d70-4ac4-b521-c5868508fb51","Type":"ContainerDied","Data":"bdd8a511fab21de0c167c89489ae065b2d5eab3dcd961be320f9148da94b9a23"}
Nov 03 19:15:04.349710 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.349689    4489 generic.go:356] "Generic (PLEG): container finished" podID="02b14985-244a-4ca7-b977-38cb452421e3" containerID="5ac9c630fc9cf41ef198ab4ce61557e76008dda9663cc0d726cef8cecafa895f" exitCode=60
Nov 03 19:15:04.349817 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.349763    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" event={"ID":"02b14985-244a-4ca7-b977-38cb452421e3","Type":"ContainerDied","Data":"5ac9c630fc9cf41ef198ab4ce61557e76008dda9663cc0d726cef8cecafa895f"}
Nov 03 19:15:04.349986 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.349970    4489 scope.go:122] "RemoveContainer" containerID="5ac9c630fc9cf41ef198ab4ce61557e76008dda9663cc0d726cef8cecafa895f"
Nov 03 19:15:04.350275 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:04.350256    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb_odh-ai-gateway-infra(02b14985-244a-4ca7-b977-38cb452421e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" podUID="02b14985-244a-4ca7-b977-38cb452421e3"
Nov 03 19:15:04.358602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.358581    4489 scope.go:122] "RemoveContainer" containerID="2be96aa3e85e5839bd303ac49757f249963bc29d4c4c10375e4f2529fe8f5306"
Nov 03 19:15:04.374098 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.374078    4489 scope.go:122] "RemoveContainer" containerID="89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834"
Nov 03 19:15:04.383803 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.383778    4489 scope.go:122] "RemoveContainer" containerID="e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b"
Nov 03 19:15:04.395561 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.395542    4489 scope.go:122] "RemoveContainer" containerID="89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834"
Nov 03 19:15:04.395923 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:04.395897    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834\": container with ID starting with 89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834 not found: ID does not exist" containerID="89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834"
Nov 03 19:15:04.396026 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.395933    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834"} err="failed to get container status \"89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834\": rpc error: code = NotFound desc = could not find container \"89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834\": container with ID starting with 89b96231a66ce0654eb95ea8f454bbda4b3a0641a4809afc46a99f948147e834 not found: ID does not exist"
Nov 03 19:15:04.396026 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.395953    4489 scope.go:122] "RemoveContainer" containerID="e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b"
Nov 03 19:15:04.396347 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:04.396323    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b\": container with ID starting with e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b not found: ID does not exist" containerID="e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b"
Nov 03 19:15:04.396467 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.396354    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b"} err="failed to get container status \"e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b\": rpc error: code = NotFound desc = could not find container \"e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b\": container with ID starting with e5667a29f5b831b20e5590594934e6f446481458b628cfec4c8fc0d73dea814b not found: ID does not exist"
Nov 03 19:15:04.396467 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.396371    4489 scope.go:122] "RemoveContainer" containerID="90769c9785285503b309fa7c5fd857147461dd682b8520ce83a56ff53b77e16e"
Nov 03 19:15:04.816656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.816606    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"]
Nov 03 19:15:04.821233 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.821210    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bf3e1c-openshift-default-b444fc74f-fpcv7"]
Nov 03 19:15:04.996822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:04.996787    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" path="/var/lib/kubelet/pods/319b6ca2-9d70-4ac4-b521-c5868508fb51/volumes"
Nov 03 19:15:05.359196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:05.359132    4489 scope.go:122] "RemoveContainer" containerID="5ac9c630fc9cf41ef198ab4ce61557e76008dda9663cc0d726cef8cecafa895f"
Nov 03 19:15:05.359686 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:05.359556    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb_odh-ai-gateway-infra(02b14985-244a-4ca7-b977-38cb452421e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" podUID="02b14985-244a-4ca7-b977-38cb452421e3"
Nov 03 19:15:05.361025 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:05.361005    4489 scope.go:122] "RemoveContainer" containerID="413de92d9609680b68d426e74b4ee53b82a2c93eae017ea1afc604b92dfe5d8b"
Nov 03 19:15:05.361381 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:05.361348    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz_odh-ai-gateway-infra(93065652-85d3-4d61-94dd-af4cdabc7651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" podUID="93065652-85d3-4d61-94dd-af4cdabc7651"
Nov 03 19:15:05.362786 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:05.362763    4489 scope.go:122] "RemoveContainer" containerID="d047179dceb071b7e8c4831406c9fd3dfae0c7234ddea55d4c3b7891339c2ad0"
Nov 03 19:15:05.363095 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:05.363072    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr_odh-ai-gateway-infra(4c1d06d1-840f-4e44-8740-f42b804dae84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84"
Nov 03 19:15:13.988511 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:13.988483    4489 scope.go:122] "RemoveContainer" containerID="413de92d9609680b68d426e74b4ee53b82a2c93eae017ea1afc604b92dfe5d8b"
Nov 03 19:15:13.988994 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:13.988662    4489 scope.go:122] "RemoveContainer" containerID="21b198b921264a15c00881b688153368f0918963d1b7c2c020b6085cddab5bf8"
Nov 03 19:15:14.553838 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.553759    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"]
Nov 03 19:15:14.751258 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.751226    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:14.858887 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.858808    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1d06d1-840f-4e44-8740-f42b804dae84-kube-api-access-xr54b\" (UniqueName: \"kubernetes.io/projected/4c1d06d1-840f-4e44-8740-f42b804dae84-kube-api-access-xr54b\") pod \"4c1d06d1-840f-4e44-8740-f42b804dae84\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") "
Nov 03 19:15:14.859032 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.858980    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1d06d1-840f-4e44-8740-f42b804dae84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d06d1-840f-4e44-8740-f42b804dae84-maas-api-service-ca\") pod \"4c1d06d1-840f-4e44-8740-f42b804dae84\" (UID: \"4c1d06d1-840f-4e44-8740-f42b804dae84\") "
Nov 03 19:15:14.859351 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.859328    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1d06d1-840f-4e44-8740-f42b804dae84-maas-api-service-ca" pod "4c1d06d1-840f-4e44-8740-f42b804dae84" (UID: "4c1d06d1-840f-4e44-8740-f42b804dae84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:14.861051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.861029    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1d06d1-840f-4e44-8740-f42b804dae84-kube-api-access-xr54b" pod "4c1d06d1-840f-4e44-8740-f42b804dae84" (UID: "4c1d06d1-840f-4e44-8740-f42b804dae84"). InnerVolumeSpecName "kube-api-access-xr54b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:14.962964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.962501    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1d06d1-840f-4e44-8740-f42b804dae84-maas-api-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:14.962964 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:14.962558    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr54b\" (UniqueName: \"kubernetes.io/projected/4c1d06d1-840f-4e44-8740-f42b804dae84-kube-api-access-xr54b\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:15.456356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.456281    4489 generic.go:356] "Generic (PLEG): container finished" podID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerID="c3909dcdd5f2d57ad56a69e469dd1d8ff9a9978c10db72a8a18d0cab304d679e" exitCode=60
Nov 03 19:15:15.457309 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.456328    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" event={"ID":"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c","Type":"ContainerDied","Data":"c3909dcdd5f2d57ad56a69e469dd1d8ff9a9978c10db72a8a18d0cab304d679e"}
Nov 03 19:15:15.457309 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.457052    4489 scope.go:122] "RemoveContainer" containerID="21b198b921264a15c00881b688153368f0918963d1b7c2c020b6085cddab5bf8"
Nov 03 19:15:15.458054 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.457990    4489 scope.go:122] "RemoveContainer" containerID="c3909dcdd5f2d57ad56a69e469dd1d8ff9a9978c10db72a8a18d0cab304d679e"
Nov 03 19:15:15.458770 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:15.458546    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895555-hmrwr_odh-ai-gateway-infra(f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c"
Nov 03 19:15:15.462474 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.462447    4489 generic.go:356] "Generic (PLEG): container finished" podID="93065652-85d3-4d61-94dd-af4cdabc7651" containerID="37d20a492247df39ac1fd1587b6ea38b885144321537b069daad02599febc37f" exitCode=60
Nov 03 19:15:15.462594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.462553    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" event={"ID":"93065652-85d3-4d61-94dd-af4cdabc7651","Type":"ContainerDied","Data":"37d20a492247df39ac1fd1587b6ea38b885144321537b069daad02599febc37f"}
Nov 03 19:15:15.463130 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.463112    4489 scope.go:122] "RemoveContainer" containerID="37d20a492247df39ac1fd1587b6ea38b885144321537b069daad02599febc37f"
Nov 03 19:15:15.463505 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:15.463482    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz_odh-ai-gateway-infra(93065652-85d3-4d61-94dd-af4cdabc7651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" podUID="93065652-85d3-4d61-94dd-af4cdabc7651"
Nov 03 19:15:15.464682 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.464660    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr" event={"ID":"4c1d06d1-840f-4e44-8740-f42b804dae84","Type":"ContainerDied","Data":"1602b746af5264dc3855f350d1009d2b5d30312ae2521f827eedb0cb880e507d"}
Nov 03 19:15:15.464863 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.464753    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"
Nov 03 19:15:15.476765 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.476745    4489 scope.go:122] "RemoveContainer" containerID="413de92d9609680b68d426e74b4ee53b82a2c93eae017ea1afc604b92dfe5d8b"
Nov 03 19:15:15.491952 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.491936    4489 scope.go:122] "RemoveContainer" containerID="d047179dceb071b7e8c4831406c9fd3dfae0c7234ddea55d4c3b7891339c2ad0"
Nov 03 19:15:15.805627 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.805538    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"]
Nov 03 19:15:15.806212 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806197    4489 cpu_manager.go:414] "containerMap: removing container" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy"
Nov 03 19:15:15.806268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806214    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy"
Nov 03 19:15:15.806268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806237    4489 cpu_manager.go:414] "containerMap: removing container" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerName="cleanup"
Nov 03 19:15:15.806268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806243    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerName="cleanup"
Nov 03 19:15:15.806268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806253    4489 cpu_manager.go:414] "containerMap: removing container" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerName="cleanup"
Nov 03 19:15:15.806268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806258    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerName="cleanup"
Nov 03 19:15:15.806426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806352    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerName="cleanup"
Nov 03 19:15:15.806426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806362    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84" containerName="cleanup"
Nov 03 19:15:15.806426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806368    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy"
Nov 03 19:15:15.806426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.806378    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy"
Nov 03 19:15:15.811044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.811020    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.820003 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.819982    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"]
Nov 03 19:15:15.823096 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.823076    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ecf40a-29895555-ll6fr"]
Nov 03 19:15:15.866632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.866603    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"]
Nov 03 19:15:15.867512 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.867496    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:15:15.888939 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.888911    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86471317-1765-4670-a922-b60d8f4ae725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.889155 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.889042    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86471317-1765-4670-a922-b60d8f4ae725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.889244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.889217    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2k2x\" (UniqueName: \"kubernetes.io/projected/86471317-1765-4670-a922-b60d8f4ae725-kube-api-access-s2k2x\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.988668 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.988636    4489 scope.go:122] "RemoveContainer" containerID="5ac9c630fc9cf41ef198ab4ce61557e76008dda9663cc0d726cef8cecafa895f"
Nov 03 19:15:15.990110 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.990083    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86471317-1765-4670-a922-b60d8f4ae725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.990245 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.990234    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2k2x\" (UniqueName: \"kubernetes.io/projected/86471317-1765-4670-a922-b60d8f4ae725-kube-api-access-s2k2x\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.990366 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.990347    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86471317-1765-4670-a922-b60d8f4ae725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.990784 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.990765    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86471317-1765-4670-a922-b60d8f4ae725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:15.992957 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:15.992936    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86471317-1765-4670-a922-b60d8f4ae725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:16.055626 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.055583    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2k2x\" (UniqueName: \"kubernetes.io/projected/86471317-1765-4670-a922-b60d8f4ae725-kube-api-access-s2k2x\") pod \"maas-api-revoke-sub-bc0f0eec-fnjvg\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:16.125254 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.125171    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:16.485925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.485880    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" event={"ID":"02b14985-244a-4ca7-b977-38cb452421e3","Type":"ContainerStarted","Data":"b1681d4d559ba4bab31472bc783e2e141ef6002d6ae7d73f1d0d8ca8ea3b1fa5"}
Nov 03 19:15:16.514671 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:15:16.514594    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86471317_1765_4670_a922_b60d8f4ae725.slice/crio-e4bcf5e2a1210671545824ae36e45a8a515e830abc4df2a6685306829e37de92 WatchSource:0}: Error finding container e4bcf5e2a1210671545824ae36e45a8a515e830abc4df2a6685306829e37de92: Status 404 returned error can't find the container with id e4bcf5e2a1210671545824ae36e45a8a515e830abc4df2a6685306829e37de92
Nov 03 19:15:16.517222 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.517195    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"]
Nov 03 19:15:16.612734 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.612167    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"]
Nov 03 19:15:16.767491 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.766246    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"]
Nov 03 19:15:16.857511 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.857417    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:16.962815 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.962781    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:16.996790 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:16.996759    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1d06d1-840f-4e44-8740-f42b804dae84" path="/var/lib/kubelet/pods/4c1d06d1-840f-4e44-8740-f42b804dae84/volumes"
Nov 03 19:15:17.038043 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.038003    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-kube-api-access-k4m95\" (UniqueName: \"kubernetes.io/projected/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-kube-api-access-k4m95\") pod \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") "
Nov 03 19:15:17.038288 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.038266    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-maas-api-service-ca\") pod \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\" (UID: \"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c\") "
Nov 03 19:15:17.038610 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.038586    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-maas-api-service-ca" pod "f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" (UID: "f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:17.039028 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.039005    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-maas-api-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:17.050885 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.050837    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-kube-api-access-k4m95" pod "f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" (UID: "f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c"). InnerVolumeSpecName "kube-api-access-k4m95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:17.140009 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.139970    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93065652-85d3-4d61-94dd-af4cdabc7651-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93065652-85d3-4d61-94dd-af4cdabc7651-maas-api-service-ca\") pod \"93065652-85d3-4d61-94dd-af4cdabc7651\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") "
Nov 03 19:15:17.140295 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.140269    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93065652-85d3-4d61-94dd-af4cdabc7651-kube-api-access-qfzxf\" (UniqueName: \"kubernetes.io/projected/93065652-85d3-4d61-94dd-af4cdabc7651-kube-api-access-qfzxf\") pod \"93065652-85d3-4d61-94dd-af4cdabc7651\" (UID: \"93065652-85d3-4d61-94dd-af4cdabc7651\") "
Nov 03 19:15:17.141003 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.140977    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4m95\" (UniqueName: \"kubernetes.io/projected/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c-kube-api-access-k4m95\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:17.141725 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.141698    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93065652-85d3-4d61-94dd-af4cdabc7651-maas-api-service-ca" pod "93065652-85d3-4d61-94dd-af4cdabc7651" (UID: "93065652-85d3-4d61-94dd-af4cdabc7651"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:17.144552 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.144526    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93065652-85d3-4d61-94dd-af4cdabc7651-kube-api-access-qfzxf" pod "93065652-85d3-4d61-94dd-af4cdabc7651" (UID: "93065652-85d3-4d61-94dd-af4cdabc7651"). InnerVolumeSpecName "kube-api-access-qfzxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:17.251106 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.251022    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzxf\" (UniqueName: \"kubernetes.io/projected/93065652-85d3-4d61-94dd-af4cdabc7651-kube-api-access-qfzxf\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:17.251334 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.251314    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93065652-85d3-4d61-94dd-af4cdabc7651-maas-api-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:17.504538 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.504445    4489 generic.go:356] "Generic (PLEG): container finished" podID="02b14985-244a-4ca7-b977-38cb452421e3" containerID="b1681d4d559ba4bab31472bc783e2e141ef6002d6ae7d73f1d0d8ca8ea3b1fa5" exitCode=60
Nov 03 19:15:17.505001 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.504534    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" event={"ID":"02b14985-244a-4ca7-b977-38cb452421e3","Type":"ContainerDied","Data":"b1681d4d559ba4bab31472bc783e2e141ef6002d6ae7d73f1d0d8ca8ea3b1fa5"}
Nov 03 19:15:17.505001 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.504620    4489 scope.go:122] "RemoveContainer" containerID="5ac9c630fc9cf41ef198ab4ce61557e76008dda9663cc0d726cef8cecafa895f"
Nov 03 19:15:17.505100 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.505082    4489 scope.go:122] "RemoveContainer" containerID="b1681d4d559ba4bab31472bc783e2e141ef6002d6ae7d73f1d0d8ca8ea3b1fa5"
Nov 03 19:15:17.505555 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:17.505513    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb_odh-ai-gateway-infra(02b14985-244a-4ca7-b977-38cb452421e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" podUID="02b14985-244a-4ca7-b977-38cb452421e3"
Nov 03 19:15:17.506676 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.506651    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg" event={"ID":"86471317-1765-4670-a922-b60d8f4ae725","Type":"ContainerStarted","Data":"75b4ea2030af8f0db1b65b8889e49660864d9279bef04f25994139485f56f6ba"}
Nov 03 19:15:17.506786 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.506695    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg" event={"ID":"86471317-1765-4670-a922-b60d8f4ae725","Type":"ContainerStarted","Data":"e4bcf5e2a1210671545824ae36e45a8a515e830abc4df2a6685306829e37de92"}
Nov 03 19:15:17.508554 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.508536    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"
Nov 03 19:15:17.508646 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.508566    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr" event={"ID":"f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c","Type":"ContainerDied","Data":"b52383244d4c014fe71593f6a3826eee5e650115ff0993527aa054ef3753db37"}
Nov 03 19:15:17.510444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.510419    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"
Nov 03 19:15:17.510541 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.510445    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz" event={"ID":"93065652-85d3-4d61-94dd-af4cdabc7651","Type":"ContainerDied","Data":"ac1133bdc59ee25c17e6bd032edd870dfa82554c21e8e466734bfe8173b4887c"}
Nov 03 19:15:17.532331 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.532309    4489 scope.go:122] "RemoveContainer" containerID="c3909dcdd5f2d57ad56a69e469dd1d8ff9a9978c10db72a8a18d0cab304d679e"
Nov 03 19:15:17.540112 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:17.540082    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:15:17.542990 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.542968    4489 scope.go:122] "RemoveContainer" containerID="37d20a492247df39ac1fd1587b6ea38b885144321537b069daad02599febc37f"
Nov 03 19:15:17.618726 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.618695    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"]
Nov 03 19:15:17.757852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.757752    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg" podStartSLOduration=2.757737963 podStartE2EDuration="2.757737963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:17.755915888 +0000 UTC m=+2246.904610458" watchObservedRunningTime="2026-11-03 19:15:17.757737963 +0000 UTC m=+2246.906432533"
Nov 03 19:15:17.876841 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.876795    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"]
Nov 03 19:15:17.880423 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.880397    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b485d8-29895555-x74gz"]
Nov 03 19:15:17.991632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.991587    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"]
Nov 03 19:15:17.995543 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:17.995517    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895555-hmrwr"]
Nov 03 19:15:18.523248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.523220    4489 generic.go:356] "Generic (PLEG): container finished" podID="86471317-1765-4670-a922-b60d8f4ae725" containerID="75b4ea2030af8f0db1b65b8889e49660864d9279bef04f25994139485f56f6ba" exitCode=0
Nov 03 19:15:18.523663 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.523300    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg" event={"ID":"86471317-1765-4670-a922-b60d8f4ae725","Type":"ContainerDied","Data":"75b4ea2030af8f0db1b65b8889e49660864d9279bef04f25994139485f56f6ba"}
Nov 03 19:15:18.678349 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.678322    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:18.706159 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.706086    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b14985-244a-4ca7-b977-38cb452421e3-kube-api-access-mtwtk\" (UniqueName: \"kubernetes.io/projected/02b14985-244a-4ca7-b977-38cb452421e3-kube-api-access-mtwtk\") pod \"02b14985-244a-4ca7-b977-38cb452421e3\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") "
Nov 03 19:15:18.706335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.706292    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b14985-244a-4ca7-b977-38cb452421e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b14985-244a-4ca7-b977-38cb452421e3-maas-api-service-ca\") pod \"02b14985-244a-4ca7-b977-38cb452421e3\" (UID: \"02b14985-244a-4ca7-b977-38cb452421e3\") "
Nov 03 19:15:18.706768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.706736    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b14985-244a-4ca7-b977-38cb452421e3-maas-api-service-ca" pod "02b14985-244a-4ca7-b977-38cb452421e3" (UID: "02b14985-244a-4ca7-b977-38cb452421e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:18.707061 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.706974    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b14985-244a-4ca7-b977-38cb452421e3-maas-api-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.708689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.708653    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b14985-244a-4ca7-b977-38cb452421e3-kube-api-access-mtwtk" pod "02b14985-244a-4ca7-b977-38cb452421e3" (UID: "02b14985-244a-4ca7-b977-38cb452421e3"). InnerVolumeSpecName "kube-api-access-mtwtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:18.807895 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.807805    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtwtk\" (UniqueName: \"kubernetes.io/projected/02b14985-244a-4ca7-b977-38cb452421e3-kube-api-access-mtwtk\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.996335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.996290    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" path="/var/lib/kubelet/pods/93065652-85d3-4d61-94dd-af4cdabc7651/volumes"
Nov 03 19:15:18.996860 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:18.996838    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" path="/var/lib/kubelet/pods/f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c/volumes"
Nov 03 19:15:19.551268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.551236    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"
Nov 03 19:15:19.551766 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.551264    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb" event={"ID":"02b14985-244a-4ca7-b977-38cb452421e3","Type":"ContainerDied","Data":"709b9b3016897cec8b9213c6337c7ecd2f6ff3f20ca81cca6c159dfe5d8864fa"}
Nov 03 19:15:19.551766 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.551328    4489 scope.go:122] "RemoveContainer" containerID="b1681d4d559ba4bab31472bc783e2e141ef6002d6ae7d73f1d0d8ca8ea3b1fa5"
Nov 03 19:15:19.740585 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.740447    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"]
Nov 03 19:15:19.745766 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.745484    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-32a59b-29895555-mwbbb"]
Nov 03 19:15:19.810751 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.810672    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:19.854327 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.853864    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86471317-1765-4670-a922-b60d8f4ae725-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86471317-1765-4670-a922-b60d8f4ae725-maas-api-xks-ca\") pod \"86471317-1765-4670-a922-b60d8f4ae725\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") "
Nov 03 19:15:19.854327 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.854108    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86471317-1765-4670-a922-b60d8f4ae725-kube-api-access-s2k2x\" (UniqueName: \"kubernetes.io/projected/86471317-1765-4670-a922-b60d8f4ae725-kube-api-access-s2k2x\") pod \"86471317-1765-4670-a922-b60d8f4ae725\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") "
Nov 03 19:15:19.854327 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.854215    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86471317-1765-4670-a922-b60d8f4ae725-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86471317-1765-4670-a922-b60d8f4ae725-maas-api-openshift-ca\") pod \"86471317-1765-4670-a922-b60d8f4ae725\" (UID: \"86471317-1765-4670-a922-b60d8f4ae725\") "
Nov 03 19:15:19.854683 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.854549    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86471317-1765-4670-a922-b60d8f4ae725-maas-api-openshift-ca" pod "86471317-1765-4670-a922-b60d8f4ae725" (UID: "86471317-1765-4670-a922-b60d8f4ae725"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:19.855059 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.855035    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86471317-1765-4670-a922-b60d8f4ae725-maas-api-openshift-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:19.856425 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.856379    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86471317-1765-4670-a922-b60d8f4ae725-maas-api-xks-ca" pod "86471317-1765-4670-a922-b60d8f4ae725" (UID: "86471317-1765-4670-a922-b60d8f4ae725"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:15:19.856999 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.856972    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86471317-1765-4670-a922-b60d8f4ae725-kube-api-access-s2k2x" pod "86471317-1765-4670-a922-b60d8f4ae725" (UID: "86471317-1765-4670-a922-b60d8f4ae725"). InnerVolumeSpecName "kube-api-access-s2k2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:19.958529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.957298    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86471317-1765-4670-a922-b60d8f4ae725-maas-api-xks-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:19.958529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:19.958054    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2k2x\" (UniqueName: \"kubernetes.io/projected/86471317-1765-4670-a922-b60d8f4ae725-kube-api-access-s2k2x\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.565491 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:20.565458    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"
Nov 03 19:15:20.565491 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:20.565475    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg" event={"ID":"86471317-1765-4670-a922-b60d8f4ae725","Type":"ContainerDied","Data":"e4bcf5e2a1210671545824ae36e45a8a515e830abc4df2a6685306829e37de92"}
Nov 03 19:15:20.566004 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:20.565533    4489 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4bcf5e2a1210671545824ae36e45a8a515e830abc4df2a6685306829e37de92"
Nov 03 19:15:21.019118 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:15:21.019083    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b14985-244a-4ca7-b977-38cb452421e3" path="/var/lib/kubelet/pods/02b14985-244a-4ca7-b977-38cb452421e3/volumes"
Nov 03 19:15:47.551941 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:15:47.551901    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:16:17.534839 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:17.534747    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:16:24.623904 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:24.623871    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"]
Nov 03 19:16:24.624395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:24.624063    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy" containerID="cri-o://6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46" gracePeriod=30
Nov 03 19:16:29.926601 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:29.926572    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:16:30.058335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.058301    4489 generic.go:356] "Generic (PLEG): container finished" podID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerID="6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46" exitCode=0
Nov 03 19:16:30.058509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.058336    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" event={"ID":"4e29bf55-1ada-4be6-a5ff-af33f146b8a4","Type":"ContainerDied","Data":"6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46"}
Nov 03 19:16:30.058509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.058407    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn" event={"ID":"4e29bf55-1ada-4be6-a5ff-af33f146b8a4","Type":"ContainerDied","Data":"9f98df4b8dca2f58ad531b8c566a7d80d5c1010bc83a5e6ecc3f8230d5cd8c30"}
Nov 03 19:16:30.058509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.058349    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"
Nov 03 19:16:30.058509 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.058439    4489 scope.go:122] "RemoveContainer" containerID="6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46"
Nov 03 19:16:30.069378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.069360    4489 scope.go:122] "RemoveContainer" containerID="6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46"
Nov 03 19:16:30.069650 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:30.069629    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46\": container with ID starting with 6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46 not found: ID does not exist" containerID="6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46"
Nov 03 19:16:30.069720 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.069662    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46"} err="failed to get container status \"6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46\": rpc error: code = NotFound desc = could not find container \"6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46\": container with ID starting with 6e320c4ecdbda607c3ec49c47cf21b6829732c06236b7da70397a7b1521cff46 not found: ID does not exist"
Nov 03 19:16:30.085241 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085216    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-kube-api-access-6cfv5\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-kube-api-access-6cfv5\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085360 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085328    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-podinfo\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085421 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085382    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-certs\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085474 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085432    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-envoy\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085591 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085571    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-token\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-token\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085727 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085623    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-data\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085727 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085663    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istiod-ca-cert\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085727 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085690    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-certs" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:30.085727 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085715    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-socket\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.085973 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085779    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-credential-socket\") pod \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\" (UID: \"4e29bf55-1ada-4be6-a5ff-af33f146b8a4\") "
Nov 03 19:16:30.086031 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.085972    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-data" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:30.086607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086228    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-credential-socket" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:30.086607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086313    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istiod-ca-cert" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:30.086607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086578    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-socket" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:30.086879 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086845    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.086879 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086866    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.086992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086882    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.086992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086896    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.086992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.086912    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.087768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.087743    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-podinfo" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:16:30.087881 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.087835    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-kube-api-access-6cfv5" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "kube-api-access-6cfv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:30.088029 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.088007    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-envoy" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:30.088137 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.088104    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-token" pod "4e29bf55-1ada-4be6-a5ff-af33f146b8a4" (UID: "4e29bf55-1ada-4be6-a5ff-af33f146b8a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:30.188209 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.188177    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.188209 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.188204    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.188209 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.188213    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.188442 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.188222    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cfv5\" (UniqueName: \"kubernetes.io/projected/4e29bf55-1ada-4be6-a5ff-af33f146b8a4-kube-api-access-6cfv5\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:30.487384 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.487301    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"]
Nov 03 19:16:30.538769 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.538735    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-t7qrn"]
Nov 03 19:16:30.996538 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:30.996505    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" path="/var/lib/kubelet/pods/4e29bf55-1ada-4be6-a5ff-af33f146b8a4/volumes"
Nov 03 19:16:37.135903 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:37.135863    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"]
Nov 03 19:16:37.136350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:37.136096    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy" containerID="cri-o://30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3" gracePeriod=30
Nov 03 19:16:39.803888 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:39.803856    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"]
Nov 03 19:16:39.804428 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:39.804220    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" containerID="cri-o://a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec" gracePeriod=30
Nov 03 19:16:39.928852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:39.928819    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"]
Nov 03 19:16:39.929045 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:39.929025    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy" containerID="cri-o://24b8769c99134aade74c9c926cf789eda4d1d095b034538c1c1822e1be759129" gracePeriod=30
Nov 03 19:16:40.796578 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:40.796538    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"]
Nov 03 19:16:40.910618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:40.910586    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"]
Nov 03 19:16:42.662198 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.662168    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.774752    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-credential-socket\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.774831    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-data\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.774880    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-podinfo\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.774921    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istiod-ca-cert\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.775005    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-token\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-token\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.775054    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-socket\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.775228    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-certs\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.775303    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-envoy\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.778378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.775399    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-kube-api-access-9t2vl\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-kube-api-access-9t2vl\") pod \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\" (UID: \"0c674b2f-1db3-4a15-9e8f-aa148d40025e\") "
Nov 03 19:16:42.779032 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.778516    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-socket" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:42.790324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.790269    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-data" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:42.792319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.792286    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-credential-socket" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:42.792759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.792733    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-certs" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:42.793642 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.793594    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-token" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:42.811559 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.811515    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istiod-ca-cert" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:42.815741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.815698    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-envoy" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:42.816011 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.815987    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-podinfo" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:16:42.816618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.816592    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-kube-api-access-9t2vl" pod "0c674b2f-1db3-4a15-9e8f-aa148d40025e" (UID: "0c674b2f-1db3-4a15-9e8f-aa148d40025e"). InnerVolumeSpecName "kube-api-access-9t2vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:42.905882 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.905777    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906121 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906098    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906275    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2vl\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-kube-api-access-9t2vl\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906419 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906406    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906535 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906522    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906653 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906640    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906764 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906751    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906862    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c674b2f-1db3-4a15-9e8f-aa148d40025e-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:42.906987 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:42.906974    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c674b2f-1db3-4a15-9e8f-aa148d40025e-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:43.219740 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.219706    4489 generic.go:356] "Generic (PLEG): container finished" podID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerID="30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3" exitCode=0
Nov 03 19:16:43.219871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.219783    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"
Nov 03 19:16:43.219871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.219787    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" event={"ID":"0c674b2f-1db3-4a15-9e8f-aa148d40025e","Type":"ContainerDied","Data":"30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3"}
Nov 03 19:16:43.219871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.219844    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g" event={"ID":"0c674b2f-1db3-4a15-9e8f-aa148d40025e","Type":"ContainerDied","Data":"a1287fb53de6a9a7f043cfb86faf537f2c6e552168bda60ba4aad7e1d408753f"}
Nov 03 19:16:43.219871 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.219864    4489 scope.go:122] "RemoveContainer" containerID="30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3"
Nov 03 19:16:43.230251 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.230219    4489 scope.go:122] "RemoveContainer" containerID="30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3"
Nov 03 19:16:43.230517 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:43.230494    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3\": container with ID starting with 30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3 not found: ID does not exist" containerID="30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3"
Nov 03 19:16:43.230565 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.230528    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3"} err="failed to get container status \"30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3\": rpc error: code = NotFound desc = could not find container \"30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3\": container with ID starting with 30995d05f757e86828aa081c4054210af2773bd5857908686d03835d225e48b3 not found: ID does not exist"
Nov 03 19:16:43.393890 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.393855    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"]
Nov 03 19:16:43.445525 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.445492    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-z2f5g"]
Nov 03 19:16:43.800110 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.800068    4489 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:16:43.800552 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:43.800179    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:16:44.997393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:44.997353    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" path="/var/lib/kubelet/pods/0c674b2f-1db3-4a15-9e8f-aa148d40025e/volumes"
Nov 03 19:16:45.139023 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.138997    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238091    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-kube-api-access-rdh94\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-kube-api-access-rdh94\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238161    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istiod-ca-cert\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238210    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-certs\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238287    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-data\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238355    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-token\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-token\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238387    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-envoy\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238490    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-podinfo\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238580    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-credential-socket\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.239184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.238660    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-socket\") pod \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\" (UID: \"fb54b5f0-0fa8-4f5b-9255-ccc89e323396\") "
Nov 03 19:16:45.241054 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.241010    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-socket" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.261536 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.261230    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-credential-socket" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.270454 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.265225    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-certs" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.271489 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.271459    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istiod-ca-cert" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:45.271889 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.271863    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-data" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.289903 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.286468    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-podinfo" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:16:45.294951 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.294904    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-token" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:45.295266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.295230    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-kube-api-access-rdh94" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "kube-api-access-rdh94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:45.306340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.306053    4489 generic.go:356] "Generic (PLEG): container finished" podID="47e4111b-5349-4747-b39d-adba3dd53310" containerID="24b8769c99134aade74c9c926cf789eda4d1d095b034538c1c1822e1be759129" exitCode=0
Nov 03 19:16:45.306340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.306184    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" event={"ID":"47e4111b-5349-4747-b39d-adba3dd53310","Type":"ContainerDied","Data":"24b8769c99134aade74c9c926cf789eda4d1d095b034538c1c1822e1be759129"}
Nov 03 19:16:45.306340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.306246    4489 scope.go:122] "RemoveContainer" containerID="61d6095d50e5835857cdf9516972885823ef2a61d364cf78926e5e7994866c15"
Nov 03 19:16:45.307414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.307294    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 19:16:45.309348 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.308856    4489 generic.go:356] "Generic (PLEG): container finished" podID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerID="a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec" exitCode=0
Nov 03 19:16:45.309348 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.308894    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" event={"ID":"fb54b5f0-0fa8-4f5b-9255-ccc89e323396","Type":"ContainerDied","Data":"a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec"}
Nov 03 19:16:45.309348 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.308938    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx" event={"ID":"fb54b5f0-0fa8-4f5b-9255-ccc89e323396","Type":"ContainerDied","Data":"3957cb4ad100b74315f6fccbd42d85bd872f97c56e8c1a0c519ade29bf0559c8"}
Nov 03 19:16:45.309348 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.309009    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"
Nov 03 19:16:45.322867 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.310294    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-envoy" pod "fb54b5f0-0fa8-4f5b-9255-ccc89e323396" (UID: "fb54b5f0-0fa8-4f5b-9255-ccc89e323396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.345441 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345349    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdh94\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-kube-api-access-rdh94\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345441 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345398    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345441 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345414    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345441 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345430    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345445    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345461    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345475    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345489    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.345741 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.345503    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb54b5f0-0fa8-4f5b-9255-ccc89e323396-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.372986 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.372847    4489 scope.go:122] "RemoveContainer" containerID="a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec"
Nov 03 19:16:45.387705 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.387676    4489 scope.go:122] "RemoveContainer" containerID="a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec"
Nov 03 19:16:45.388084 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:45.388058    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec\": container with ID starting with a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec not found: ID does not exist" containerID="a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec"
Nov 03 19:16:45.388220 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.388097    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec"} err="failed to get container status \"a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec\": rpc error: code = NotFound desc = could not find container \"a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec\": container with ID starting with a1b33fa5d4f062909eed46f8132c73bf7f3e98da20558732b7de5a2a9bb169ec not found: ID does not exist"
Nov 03 19:16:45.446876 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.446842    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e4111b-5349-4747-b39d-adba3dd53310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e4111b-5349-4747-b39d-adba3dd53310-istiod-ca-cert\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.446901    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-credential-socket\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.446969    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e4111b-5349-4747-b39d-adba3dd53310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e4111b-5349-4747-b39d-adba3dd53310-istio-podinfo\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447065    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-kube-api-access-m42mk\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-kube-api-access-m42mk\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447129    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-envoy\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447216    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-certs\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447264    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-istio-token\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-istio-token\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447298    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-credential-socket" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.447347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447308    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-socket\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447326    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e4111b-5349-4747-b39d-adba3dd53310-istiod-ca-cert" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:45.447656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447482    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-data\") pod \"47e4111b-5349-4747-b39d-adba3dd53310\" (UID: \"47e4111b-5349-4747-b39d-adba3dd53310\") "
Nov 03 19:16:45.447656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447491    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-socket" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.447656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.447649    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-certs" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.448992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.448441    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.448992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.448465    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.448992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.448480    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e4111b-5349-4747-b39d-adba3dd53310-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.448992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.448496    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.448992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.448640    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-data" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.450527 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.450503    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-kube-api-access-m42mk" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "kube-api-access-m42mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:45.451874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.451853    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-istio-token" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:45.453076 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.453046    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-envoy" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:45.453497 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.453481    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e4111b-5349-4747-b39d-adba3dd53310-istio-podinfo" pod "47e4111b-5349-4747-b39d-adba3dd53310" (UID: "47e4111b-5349-4747-b39d-adba3dd53310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:16:45.550395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.550352    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e4111b-5349-4747-b39d-adba3dd53310-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.550395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.550387    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42mk\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-kube-api-access-m42mk\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.550652 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.550405    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.550652 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.550419    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e4111b-5349-4747-b39d-adba3dd53310-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.550652 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.550433    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e4111b-5349-4747-b39d-adba3dd53310-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:45.739219 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.738083    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"]
Nov 03 19:16:45.750193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.750162    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-67lkx"]
Nov 03 19:16:45.797900 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.797845    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd" podUID="97ddfc62-ffe9-4e3b-aeee-44aea01eef58" containerName="payload-processing" containerID="cri-o://cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097" gracePeriod=25
Nov 03 19:16:45.911049 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:45.911013    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27" podUID="65eb0dc6-a873-437e-a958-8a48edbc2158" containerName="payload-pre-processing" containerID="cri-o://83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e" gracePeriod=25
Nov 03 19:16:46.098624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.098591    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:16:46.264778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.263716    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-plugins-config-volume\") pod \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") "
Nov 03 19:16:46.264778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.263865    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-kube-api-access-gr4j4\" (UniqueName: \"kubernetes.io/projected/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-kube-api-access-gr4j4\") pod \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\" (UID: \"97ddfc62-ffe9-4e3b-aeee-44aea01eef58\") "
Nov 03 19:16:46.267125 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.267021    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-plugins-config-volume" pod "97ddfc62-ffe9-4e3b-aeee-44aea01eef58" (UID: "97ddfc62-ffe9-4e3b-aeee-44aea01eef58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:46.270754 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.270720    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-kube-api-access-gr4j4" pod "97ddfc62-ffe9-4e3b-aeee-44aea01eef58" (UID: "97ddfc62-ffe9-4e3b-aeee-44aea01eef58"). InnerVolumeSpecName "kube-api-access-gr4j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:46.276736 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.276707    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:16:46.319102 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.319069    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx" event={"ID":"47e4111b-5349-4747-b39d-adba3dd53310","Type":"ContainerDied","Data":"d69399cc228685d0afe70d0e94f71ede04ee65bf4e4af0ba1eaa5b816ff92761"}
Nov 03 19:16:46.319297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.319080    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"
Nov 03 19:16:46.319297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.319136    4489 scope.go:122] "RemoveContainer" containerID="24b8769c99134aade74c9c926cf789eda4d1d095b034538c1c1822e1be759129"
Nov 03 19:16:46.320362 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.320344    4489 generic.go:356] "Generic (PLEG): container finished" podID="65eb0dc6-a873-437e-a958-8a48edbc2158" containerID="83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e" exitCode=0
Nov 03 19:16:46.320447 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.320428    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27" event={"ID":"65eb0dc6-a873-437e-a958-8a48edbc2158","Type":"ContainerDied","Data":"83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e"}
Nov 03 19:16:46.320493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.320476    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27" event={"ID":"65eb0dc6-a873-437e-a958-8a48edbc2158","Type":"ContainerDied","Data":"53ce5f8adda576ce0bc26d4515b2ba48ecd467b8a2535f9d56d7ad9e5b5c842b"}
Nov 03 19:16:46.320550 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.320538    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"
Nov 03 19:16:46.323364 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.323341    4489 generic.go:356] "Generic (PLEG): container finished" podID="97ddfc62-ffe9-4e3b-aeee-44aea01eef58" containerID="cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097" exitCode=0
Nov 03 19:16:46.323540 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.323525    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd" event={"ID":"97ddfc62-ffe9-4e3b-aeee-44aea01eef58","Type":"ContainerDied","Data":"cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097"}
Nov 03 19:16:46.323588 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.323569    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd" event={"ID":"97ddfc62-ffe9-4e3b-aeee-44aea01eef58","Type":"ContainerDied","Data":"5e04a79fa29b6c9c61b6ee6d653098616c3fd8daf15ebb8b154b96a16184c330"}
Nov 03 19:16:46.323654 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.323630    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"
Nov 03 19:16:46.330103 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.330080    4489 scope.go:122] "RemoveContainer" containerID="83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e"
Nov 03 19:16:46.340113 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.340095    4489 scope.go:122] "RemoveContainer" containerID="83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e"
Nov 03 19:16:46.340470 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:46.340452    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e\": container with ID starting with 83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e not found: ID does not exist" containerID="83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e"
Nov 03 19:16:46.340521 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.340479    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e"} err="failed to get container status \"83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e\": rpc error: code = NotFound desc = could not find container \"83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e\": container with ID starting with 83f4610d11e3585a38b36f2687079004c7b3e86c3119cb50b35ce0cfad2d9b7e not found: ID does not exist"
Nov 03 19:16:46.340521 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.340495    4489 scope.go:122] "RemoveContainer" containerID="cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097"
Nov 03 19:16:46.351442 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.351421    4489 scope.go:122] "RemoveContainer" containerID="cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097"
Nov 03 19:16:46.351706 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:46.351688    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097\": container with ID starting with cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097 not found: ID does not exist" containerID="cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097"
Nov 03 19:16:46.351784 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.351718    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097"} err="failed to get container status \"cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097\": rpc error: code = NotFound desc = could not find container \"cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097\": container with ID starting with cb591a8d7dcbef2f4dadd0bff1f07df6cca8f29c34dfbb16c91f316e9417e097 not found: ID does not exist"
Nov 03 19:16:46.364837 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.364813    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65eb0dc6-a873-437e-a958-8a48edbc2158-kube-api-access-vgqps\" (UniqueName: \"kubernetes.io/projected/65eb0dc6-a873-437e-a958-8a48edbc2158-kube-api-access-vgqps\") pod \"65eb0dc6-a873-437e-a958-8a48edbc2158\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") "
Nov 03 19:16:46.365067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.365051    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65eb0dc6-a873-437e-a958-8a48edbc2158-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eb0dc6-a873-437e-a958-8a48edbc2158-plugins-config-volume\") pod \"65eb0dc6-a873-437e-a958-8a48edbc2158\" (UID: \"65eb0dc6-a873-437e-a958-8a48edbc2158\") "
Nov 03 19:16:46.365484 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.365462    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65eb0dc6-a873-437e-a958-8a48edbc2158-plugins-config-volume" pod "65eb0dc6-a873-437e-a958-8a48edbc2158" (UID: "65eb0dc6-a873-437e-a958-8a48edbc2158"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:46.365585 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.365570    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4j4\" (UniqueName: \"kubernetes.io/projected/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-kube-api-access-gr4j4\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:46.365644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.365588    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eb0dc6-a873-437e-a958-8a48edbc2158-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:46.365644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.365597    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ddfc62-ffe9-4e3b-aeee-44aea01eef58-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:46.367329 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.367306    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65eb0dc6-a873-437e-a958-8a48edbc2158-kube-api-access-vgqps" pod "65eb0dc6-a873-437e-a958-8a48edbc2158" (UID: "65eb0dc6-a873-437e-a958-8a48edbc2158"). InnerVolumeSpecName "kube-api-access-vgqps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:46.466637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.466606    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgqps\" (UniqueName: \"kubernetes.io/projected/65eb0dc6-a873-437e-a958-8a48edbc2158-kube-api-access-vgqps\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:46.514913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.514846    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"]
Nov 03 19:16:46.519661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.519629    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-65e151-7cfcd654b7-rg6rd"]
Nov 03 19:16:46.705193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.705123    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"]
Nov 03 19:16:46.708836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.708805    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-s4jqx"]
Nov 03 19:16:46.883344 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.883261    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"]
Nov 03 19:16:46.887969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.887937    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-65e151-74c97f7547-dzk27"]
Nov 03 19:16:46.996641 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.996606    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e4111b-5349-4747-b39d-adba3dd53310" path="/var/lib/kubelet/pods/47e4111b-5349-4747-b39d-adba3dd53310/volumes"
Nov 03 19:16:46.997006 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.996990    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65eb0dc6-a873-437e-a958-8a48edbc2158" path="/var/lib/kubelet/pods/65eb0dc6-a873-437e-a958-8a48edbc2158/volumes"
Nov 03 19:16:46.997374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.997359    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ddfc62-ffe9-4e3b-aeee-44aea01eef58" path="/var/lib/kubelet/pods/97ddfc62-ffe9-4e3b-aeee-44aea01eef58/volumes"
Nov 03 19:16:46.998086 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:46.998067    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" path="/var/lib/kubelet/pods/fb54b5f0-0fa8-4f5b-9255-ccc89e323396/volumes"
Nov 03 19:16:47.533798 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:47.533657    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:16:47.714268 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:47.714232    4489 patch_prober.go:29] interesting pod/e2e-inf-a-65e151-openshift-default-5478585457-cxqss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:16:47.714428 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:47.714288    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:16:49.352374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.352288    4489 generic.go:356] "Generic (PLEG): container finished" podID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerID="693969bc77b65c4e37f37dc60510341943f1089a841b619bd4892dbb33b9760f" exitCode=0
Nov 03 19:16:49.352374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.352314    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" event={"ID":"8c251c64-ee71-4e62-99cc-40ca3e911ab1","Type":"ContainerDied","Data":"693969bc77b65c4e37f37dc60510341943f1089a841b619bd4892dbb33b9760f"}
Nov 03 19:16:49.352933 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.352704    4489 scope.go:122] "RemoveContainer" containerID="693969bc77b65c4e37f37dc60510341943f1089a841b619bd4892dbb33b9760f"
Nov 03 19:16:49.354335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.354308    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:16:49.354426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.354380    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:16:49.354426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.354417    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:16:49.463859 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.463830    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"]
Nov 03 19:16:49.464118 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.464088    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy" containerID="cri-o://de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579" gracePeriod=30
Nov 03 19:16:49.514285 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.514256    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"]
Nov 03 19:16:49.514504 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:49.514481    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy" containerID="cri-o://a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814" gracePeriod=30
Nov 03 19:16:50.379225 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:50.379185    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" event={"ID":"8c251c64-ee71-4e62-99cc-40ca3e911ab1","Type":"ContainerStarted","Data":"6ec4a80033addd1c4dae59953a62a12e8b876c79adabb710629f1fdc26f32f3f"}
Nov 03 19:16:51.020921 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:51.020826    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:16:51.021564 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:51.021533    4489 patch_prober.go:29] interesting pod/e2e-inf-a-65e151-openshift-default-5478585457-cxqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:16:51.021768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:51.021743    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:16:52.005780 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:52.005744    4489 patch_prober.go:29] interesting pod/e2e-inf-a-65e151-openshift-default-5478585457-cxqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:16:52.006210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:52.005799    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:16:53.006232 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:53.006199    4489 patch_prober.go:29] interesting pod/e2e-inf-a-65e151-openshift-default-5478585457-cxqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:16:53.006717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:53.006254    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:16:54.006253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.006218    4489 patch_prober.go:29] interesting pod/e2e-inf-a-65e151-openshift-default-5478585457-cxqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:16:54.006667 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.006307    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:16:54.816562 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.816537    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:16:54.879946 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.879915    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-envoy\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.879964    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-kube-api-access-m79xc\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-kube-api-access-m79xc\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880019    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-certs\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880064    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-credential-socket\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880120    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c55759-2371-4366-9749-b11edeb3a7dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c55759-2371-4366-9749-b11edeb3a7dc-istiod-ca-cert\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880385 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880260    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-token\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880385 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880293    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-socket\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880385 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880342    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-podinfo\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880385 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880349    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-certs" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:54.880594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880454    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-credential-socket" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:54.880594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880511    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-data\") pod \"a7c55759-2371-4366-9749-b11edeb3a7dc\" (UID: \"a7c55759-2371-4366-9749-b11edeb3a7dc\") "
Nov 03 19:16:54.880594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880550    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-socket" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:54.880857 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880746    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c55759-2371-4366-9749-b11edeb3a7dc-istiod-ca-cert" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:54.880857 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.880809    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-data" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:54.881201 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.881180    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.881289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.881210    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.881289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.881227    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.881289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.881242    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c55759-2371-4366-9749-b11edeb3a7dc-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.881289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.881260    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.883246 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.883220    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-token" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:54.883438 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.883419    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-envoy" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:54.883731 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.883703    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-podinfo" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:16:54.885258 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.885239    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-kube-api-access-m79xc" pod "a7c55759-2371-4366-9749-b11edeb3a7dc" (UID: "a7c55759-2371-4366-9749-b11edeb3a7dc"). InnerVolumeSpecName "kube-api-access-m79xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:54.982925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.982892    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.982925 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.982925    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m79xc\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-kube-api-access-m79xc\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.983111 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.982936    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:54.983111 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:54.982944    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c55759-2371-4366-9749-b11edeb3a7dc-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.007203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.005589    4489 patch_prober.go:29] interesting pod/e2e-inf-a-65e151-openshift-default-5478585457-cxqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:16:55.007203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.005634    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 19:16:55.177617 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.177583    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:16:55.286749 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286718    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-socket\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.286940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286771    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-certs\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.286940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286802    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-data\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.286940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286863    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a10f023-91be-4a7b-b2c8-2ff070481e84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10f023-91be-4a7b-b2c8-2ff070481e84-istiod-ca-cert\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.286940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286889    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-envoy\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.286940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286910    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-kube-api-access-2fm57\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-kube-api-access-2fm57\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.286940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286943    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-credential-socket\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.287338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.286992    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-socket" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:55.287338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.287013    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-podinfo\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.287338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.287209    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-certs" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:55.287338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.287225    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-data" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:55.287338 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.287256    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a10f023-91be-4a7b-b2c8-2ff070481e84-istiod-ca-cert" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:16:55.287912 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.287355    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-credential-socket" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:55.287912 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.287364    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-token\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-token\") pod \"5a10f023-91be-4a7b-b2c8-2ff070481e84\" (UID: \"5a10f023-91be-4a7b-b2c8-2ff070481e84\") "
Nov 03 19:16:55.288210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.287971    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.288210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.288043    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.288210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.288072    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.288210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.288087    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a10f023-91be-4a7b-b2c8-2ff070481e84-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.288210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.288102    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.289884 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.289858    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-kube-api-access-2fm57" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "kube-api-access-2fm57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:55.289884 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.289854    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-envoy" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:16:55.290093 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.290072    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-podinfo" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:16:55.290196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.290129    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-token" pod "5a10f023-91be-4a7b-b2c8-2ff070481e84" (UID: "5a10f023-91be-4a7b-b2c8-2ff070481e84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:16:55.389892 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.389796    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.389892 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.389832    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.389892 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.389841    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a10f023-91be-4a7b-b2c8-2ff070481e84-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.389892 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.389851    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fm57\" (UniqueName: \"kubernetes.io/projected/5a10f023-91be-4a7b-b2c8-2ff070481e84-kube-api-access-2fm57\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:16:55.414417 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.414384    4489 generic.go:356] "Generic (PLEG): container finished" podID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerID="a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814" exitCode=0
Nov 03 19:16:55.414594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.414432    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"
Nov 03 19:16:55.414594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.414474    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" event={"ID":"a7c55759-2371-4366-9749-b11edeb3a7dc","Type":"ContainerDied","Data":"a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814"}
Nov 03 19:16:55.414594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.414538    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9" event={"ID":"a7c55759-2371-4366-9749-b11edeb3a7dc","Type":"ContainerDied","Data":"c61fcbacb98b815da3a6715b7318131735ff88c79ea3d7ad14069c7dd3200836"}
Nov 03 19:16:55.414755 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.414571    4489 scope.go:122] "RemoveContainer" containerID="a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814"
Nov 03 19:16:55.416320 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.416060    4489 generic.go:356] "Generic (PLEG): container finished" podID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerID="de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579" exitCode=0
Nov 03 19:16:55.416320 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.416178    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"
Nov 03 19:16:55.416320 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.416245    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" event={"ID":"5a10f023-91be-4a7b-b2c8-2ff070481e84","Type":"ContainerDied","Data":"de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579"}
Nov 03 19:16:55.416320 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.416286    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw" event={"ID":"5a10f023-91be-4a7b-b2c8-2ff070481e84","Type":"ContainerDied","Data":"00bc58f0350c76b230b217062eee9f86cbb18815d906fe9dbea43b87bbfdf827"}
Nov 03 19:16:55.436060 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.436039    4489 scope.go:122] "RemoveContainer" containerID="a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814"
Nov 03 19:16:55.436430 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:55.436397    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814\": container with ID starting with a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814 not found: ID does not exist" containerID="a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814"
Nov 03 19:16:55.436548 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.436440    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814"} err="failed to get container status \"a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814\": rpc error: code = NotFound desc = could not find container \"a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814\": container with ID starting with a34dccbe4b90fe9487cb67b4ba2da553ed5632fee1ada002f7eb792e9dde0814 not found: ID does not exist"
Nov 03 19:16:55.436548 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.436466    4489 scope.go:122] "RemoveContainer" containerID="de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579"
Nov 03 19:16:55.447407 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.447383    4489 scope.go:122] "RemoveContainer" containerID="de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579"
Nov 03 19:16:55.447665 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:16:55.447645    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579\": container with ID starting with de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579 not found: ID does not exist" containerID="de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579"
Nov 03 19:16:55.447720 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.447673    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579"} err="failed to get container status \"de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579\": rpc error: code = NotFound desc = could not find container \"de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579\": container with ID starting with de8877638ca5ab3ab7fe0f7b8c90cfd03032108557cc63664b646a3333904579 not found: ID does not exist"
Nov 03 19:16:55.588986 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.588902    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"]
Nov 03 19:16:55.593075 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.593051    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-jdpnw"]
Nov 03 19:16:55.705774 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.705738    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"]
Nov 03 19:16:55.711626 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:55.711603    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-dn2p9"]
Nov 03 19:16:56.010261 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:56.010185    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:16:56.010619 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:56.010601    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:16:56.011345 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:56.011322    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:16:56.995559 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:56.995531    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" path="/var/lib/kubelet/pods/5a10f023-91be-4a7b-b2c8-2ff070481e84/volumes"
Nov 03 19:16:56.995983 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:16:56.995969    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" path="/var/lib/kubelet/pods/a7c55759-2371-4366-9749-b11edeb3a7dc/volumes"
Nov 03 19:17:17.532688 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:17:17.532656    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:17:28.108167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:17:28.107320    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-v7mn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:17:28.108167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:17:28.107398    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-v7mn2" podUID="6ddb111d-2ea5-43a9-8362-1122f85f28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:17:28.194613 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:17:28.194570    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:17:28.194953 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:17:28.194918    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:17:28.370347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:17:28.370266    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:17:28.370636 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:17:28.370599    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:17:47.536492 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:17:47.536452    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:18:00.729659 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:00.729626    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"]
Nov 03 19:18:00.730210 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:00.729885    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy" containerID="cri-o://6ec4a80033addd1c4dae59953a62a12e8b876c79adabb710629f1fdc26f32f3f" gracePeriod=30
Nov 03 19:18:05.882030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:05.881996    4489 generic.go:356] "Generic (PLEG): container finished" podID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerID="6ec4a80033addd1c4dae59953a62a12e8b876c79adabb710629f1fdc26f32f3f" exitCode=0
Nov 03 19:18:05.882431 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:05.882075    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" event={"ID":"8c251c64-ee71-4e62-99cc-40ca3e911ab1","Type":"ContainerDied","Data":"6ec4a80033addd1c4dae59953a62a12e8b876c79adabb710629f1fdc26f32f3f"}
Nov 03 19:18:05.882431 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:05.882171    4489 scope.go:122] "RemoveContainer" containerID="693969bc77b65c4e37f37dc60510341943f1089a841b619bd4892dbb33b9760f"
Nov 03 19:18:05.989475 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:05.989444    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:18:06.020327 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020266    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-podinfo\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020439 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020353    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-token\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-token\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020439 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020386    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istiod-ca-cert\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020513 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020468    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-envoy\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020554 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020524    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-credential-socket\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020609 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020591    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-certs\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020669 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020624    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-data\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020728 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020691    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-socket\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020784 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020764    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-kube-api-access-gt5df\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-kube-api-access-gt5df\") pod \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\" (UID: \"8c251c64-ee71-4e62-99cc-40ca3e911ab1\") "
Nov 03 19:18:06.020943 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020848    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-certs" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:06.020943 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020864    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-credential-socket" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:06.020943 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.020875    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istiod-ca-cert" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:06.021198 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.021067    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-socket" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:06.021198 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.021097    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-data" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:06.021752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.021688    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.021752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.021720    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.021752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.021737    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.021752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.021752    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.021752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.021766    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.022687 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.022663    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-podinfo" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:18:06.022821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.022806    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-token" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:06.022892 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.022872    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-envoy" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:06.023297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.023278    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-kube-api-access-gt5df" pod "8c251c64-ee71-4e62-99cc-40ca3e911ab1" (UID: "8c251c64-ee71-4e62-99cc-40ca3e911ab1"). InnerVolumeSpecName "kube-api-access-gt5df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:06.123051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.123015    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.123051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.123052    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5df\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-kube-api-access-gt5df\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.123318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.123069    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.123318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.123081    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c251c64-ee71-4e62-99cc-40ca3e911ab1-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:06.889498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.889461    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss" event={"ID":"8c251c64-ee71-4e62-99cc-40ca3e911ab1","Type":"ContainerDied","Data":"bb3cab591fd2d151f4d9b822787f6c0a641158e5af365f2d5802a53e132544f3"}
Nov 03 19:18:06.889498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.889484    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"
Nov 03 19:18:06.890045 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:06.889523    4489 scope.go:122] "RemoveContainer" containerID="6ec4a80033addd1c4dae59953a62a12e8b876c79adabb710629f1fdc26f32f3f"
Nov 03 19:18:07.115136 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:07.115102    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"]
Nov 03 19:18:07.166752 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:07.166728    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-65e151-openshift-default-5478585457-cxqss"]
Nov 03 19:18:07.585645 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:07.585565    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"]
Nov 03 19:18:07.599303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:07.599275    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"]
Nov 03 19:18:08.995362 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:08.995334    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" path="/var/lib/kubelet/pods/8c251c64-ee71-4e62-99cc-40ca3e911ab1/volumes"
Nov 03 19:18:14.021064 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:14.021028    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"]
Nov 03 19:18:16.502903 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:16.502873    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"]
Nov 03 19:18:16.613929 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:16.613890    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"]
Nov 03 19:18:17.531098 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:17.531064    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:18:17.648910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.648873    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"]
Nov 03 19:18:17.649112 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.649076    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh" podUID="44f00105-4d75-443d-bf04-5394b9953c34" containerName="maas-api" containerID="cri-o://4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919" gracePeriod=30
Nov 03 19:18:17.924228 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.924205    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:18:17.962982 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.962953    4489 generic.go:356] "Generic (PLEG): container finished" podID="44f00105-4d75-443d-bf04-5394b9953c34" containerID="4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919" exitCode=0
Nov 03 19:18:17.963119 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.962994    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"
Nov 03 19:18:17.963119 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.962993    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh" event={"ID":"44f00105-4d75-443d-bf04-5394b9953c34","Type":"ContainerDied","Data":"4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919"}
Nov 03 19:18:17.963119 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.963077    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh" event={"ID":"44f00105-4d75-443d-bf04-5394b9953c34","Type":"ContainerDied","Data":"4082ed7b843602876f8000ea9052d1179c99a247e639f4560c2ebd53668763d7"}
Nov 03 19:18:17.963119 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.963111    4489 scope.go:122] "RemoveContainer" containerID="4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919"
Nov 03 19:18:17.972940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.972924    4489 scope.go:122] "RemoveContainer" containerID="4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919"
Nov 03 19:18:17.973249 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:17.973230    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919\": container with ID starting with 4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919 not found: ID does not exist" containerID="4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919"
Nov 03 19:18:17.973312 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:17.973258    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919"} err="failed to get container status \"4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919\": rpc error: code = NotFound desc = could not find container \"4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919\": container with ID starting with 4dd3586b17c7eae14dd7ee535e9d1502b6e2d9ae14012246a3e4d1a0c0009919 not found: ID does not exist"
Nov 03 19:18:18.077840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.077731    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f00105-4d75-443d-bf04-5394b9953c34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f00105-4d75-443d-bf04-5394b9953c34-openshift-service-ca\") pod \"44f00105-4d75-443d-bf04-5394b9953c34\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") "
Nov 03 19:18:18.077840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.077789    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-metrics-certs\") pod \"44f00105-4d75-443d-bf04-5394b9953c34\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") "
Nov 03 19:18:18.078109 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.077841    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls\") pod \"44f00105-4d75-443d-bf04-5394b9953c34\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") "
Nov 03 19:18:18.078109 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.077910    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f00105-4d75-443d-bf04-5394b9953c34-kube-api-access-f2fbk\" (UniqueName: \"kubernetes.io/projected/44f00105-4d75-443d-bf04-5394b9953c34-kube-api-access-f2fbk\") pod \"44f00105-4d75-443d-bf04-5394b9953c34\" (UID: \"44f00105-4d75-443d-bf04-5394b9953c34\") "
Nov 03 19:18:18.082560 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.082527    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f00105-4d75-443d-bf04-5394b9953c34-openshift-service-ca" pod "44f00105-4d75-443d-bf04-5394b9953c34" (UID: "44f00105-4d75-443d-bf04-5394b9953c34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:18.083488 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.083458    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-metrics-certs" pod "44f00105-4d75-443d-bf04-5394b9953c34" (UID: "44f00105-4d75-443d-bf04-5394b9953c34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:18.083591 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.083553    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls" pod "44f00105-4d75-443d-bf04-5394b9953c34" (UID: "44f00105-4d75-443d-bf04-5394b9953c34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:18.088600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.088568    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f00105-4d75-443d-bf04-5394b9953c34-kube-api-access-f2fbk" pod "44f00105-4d75-443d-bf04-5394b9953c34" (UID: "44f00105-4d75-443d-bf04-5394b9953c34"). InnerVolumeSpecName "kube-api-access-f2fbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:18.180074 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.180038    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f00105-4d75-443d-bf04-5394b9953c34-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:18.180074 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.180070    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:18.180074 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.180079    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f00105-4d75-443d-bf04-5394b9953c34-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:18.180367 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.180089    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2fbk\" (UniqueName: \"kubernetes.io/projected/44f00105-4d75-443d-bf04-5394b9953c34-kube-api-access-f2fbk\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:18.391302 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.391219    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"]
Nov 03 19:18:18.395875 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.395846    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-efd81c-56b66c4c49-l7smh"]
Nov 03 19:18:18.995538 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:18.995506    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f00105-4d75-443d-bf04-5394b9953c34" path="/var/lib/kubelet/pods/44f00105-4d75-443d-bf04-5394b9953c34/volumes"
Nov 03 19:18:19.022343 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:19.022299    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" podUID="1bec70e8-d473-4799-ac6d-79fa07e44c99" containerName="payload-pre-processing" containerID="cri-o://a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b" gracePeriod=25
Nov 03 19:18:21.503981 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.503938    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw" podUID="2d52a40e-0e1d-48ef-91c2-bbf189492e1b" containerName="payload-processing" containerID="cri-o://300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057" gracePeriod=25
Nov 03 19:18:21.615083 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.615045    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq" podUID="3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" containerName="payload-pre-processing" containerID="cri-o://cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4" gracePeriod=25
Nov 03 19:18:21.811035 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.811005    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:18:21.822214 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.822186    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-plugins-config-volume\") pod \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") "
Nov 03 19:18:21.822361 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.822319    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-kube-api-access-lbnpz\" (UniqueName: \"kubernetes.io/projected/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-kube-api-access-lbnpz\") pod \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\" (UID: \"2d52a40e-0e1d-48ef-91c2-bbf189492e1b\") "
Nov 03 19:18:21.822629 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.822598    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-plugins-config-volume" pod "2d52a40e-0e1d-48ef-91c2-bbf189492e1b" (UID: "2d52a40e-0e1d-48ef-91c2-bbf189492e1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:21.822919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.822899    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:21.826180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.825284    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-kube-api-access-lbnpz" pod "2d52a40e-0e1d-48ef-91c2-bbf189492e1b" (UID: "2d52a40e-0e1d-48ef-91c2-bbf189492e1b"). InnerVolumeSpecName "kube-api-access-lbnpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:21.872230 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.872203    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:18:21.923418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.923381    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-plugins-config-volume\") pod \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") "
Nov 03 19:18:21.923598 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.923486    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-kube-api-access-jfqzq\" (UniqueName: \"kubernetes.io/projected/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-kube-api-access-jfqzq\") pod \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\" (UID: \"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61\") "
Nov 03 19:18:21.923809 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.923775    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-plugins-config-volume" pod "3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" (UID: "3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:21.923950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.923933    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnpz\" (UniqueName: \"kubernetes.io/projected/2d52a40e-0e1d-48ef-91c2-bbf189492e1b-kube-api-access-lbnpz\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:21.924010 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.923956    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:21.925822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.925792    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-kube-api-access-jfqzq" pod "3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" (UID: "3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61"). InnerVolumeSpecName "kube-api-access-jfqzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:21.987943 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.987912    4489 generic.go:356] "Generic (PLEG): container finished" podID="2d52a40e-0e1d-48ef-91c2-bbf189492e1b" containerID="300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057" exitCode=0
Nov 03 19:18:21.988111 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.987950    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"
Nov 03 19:18:21.988111 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.987989    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw" event={"ID":"2d52a40e-0e1d-48ef-91c2-bbf189492e1b","Type":"ContainerDied","Data":"300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057"}
Nov 03 19:18:21.988111 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.988042    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw" event={"ID":"2d52a40e-0e1d-48ef-91c2-bbf189492e1b","Type":"ContainerDied","Data":"82a671e311e42406868c54ac0bf7175419f92e7bbd14124c6eddc1d31aecc6f9"}
Nov 03 19:18:21.988111 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.988070    4489 scope.go:122] "RemoveContainer" containerID="300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057"
Nov 03 19:18:21.989456 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.989433    4489 generic.go:356] "Generic (PLEG): container finished" podID="3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" containerID="cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4" exitCode=0
Nov 03 19:18:21.989584 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.989468    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"
Nov 03 19:18:21.989584 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.989482    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq" event={"ID":"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61","Type":"ContainerDied","Data":"cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4"}
Nov 03 19:18:21.989584 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.989517    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq" event={"ID":"3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61","Type":"ContainerDied","Data":"55309a8d7529aa178b17a0c3e49e5e5caa821dcebaeb01277338533e47ed3c9e"}
Nov 03 19:18:21.999419 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.999371    4489 scope.go:122] "RemoveContainer" containerID="300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057"
Nov 03 19:18:21.999689 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:21.999667    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057\": container with ID starting with 300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057 not found: ID does not exist" containerID="300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057"
Nov 03 19:18:21.999760 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.999696    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057"} err="failed to get container status \"300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057\": rpc error: code = NotFound desc = could not find container \"300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057\": container with ID starting with 300c0a42b58fa814e9883d2795c2523fd7109601b9284faeb2ca1040e3812057 not found: ID does not exist"
Nov 03 19:18:21.999760 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:21.999719    4489 scope.go:122] "RemoveContainer" containerID="cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4"
Nov 03 19:18:22.008213 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.008196    4489 scope.go:122] "RemoveContainer" containerID="cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4"
Nov 03 19:18:22.008470 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:22.008451    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4\": container with ID starting with cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4 not found: ID does not exist" containerID="cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4"
Nov 03 19:18:22.008507 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.008478    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4"} err="failed to get container status \"cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4\": rpc error: code = NotFound desc = could not find container \"cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4\": container with ID starting with cd8d503377f5e06ce69054ef61be963b490228c7cf78d29e4db84380f2f494a4 not found: ID does not exist"
Nov 03 19:18:22.025594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.025567    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfqzq\" (UniqueName: \"kubernetes.io/projected/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61-kube-api-access-jfqzq\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:22.169568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.169532    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"]
Nov 03 19:18:22.176110 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.176084    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-69e2b9-558bf74655-wtjfw"]
Nov 03 19:18:22.292653 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.292619    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"]
Nov 03 19:18:22.296096 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.296075    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-69e2b9-66b5bf4547-4xrkq"]
Nov 03 19:18:22.606732 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.606697    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main" containerID="cri-o://d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7" gracePeriod=45
Nov 03 19:18:22.619280 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.619250    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main" containerID="cri-o://06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a" gracePeriod=45
Nov 03 19:18:22.914829 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.914805    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:18:22.917821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.917791    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:18:22.935098 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935074    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-model-cache\") pod \"9c2859a5-32ed-4267-8838-3430edc17c3c\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") "
Nov 03 19:18:22.935189 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935124    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/635f4eda-d081-47bc-9f53-25559992ea53-tls-certs\" (UniqueName: \"kubernetes.io/secret/635f4eda-d081-47bc-9f53-25559992ea53-tls-certs\") pod \"635f4eda-d081-47bc-9f53-25559992ea53\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") "
Nov 03 19:18:22.935189 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935165    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-dshm\") pod \"9c2859a5-32ed-4267-8838-3430edc17c3c\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") "
Nov 03 19:18:22.935189 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935182    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-dshm\") pod \"635f4eda-d081-47bc-9f53-25559992ea53\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") "
Nov 03 19:18:22.935298 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935237    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2859a5-32ed-4267-8838-3430edc17c3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2859a5-32ed-4267-8838-3430edc17c3c-tls-certs\") pod \"9c2859a5-32ed-4267-8838-3430edc17c3c\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") "
Nov 03 19:18:22.935298 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935266    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635f4eda-d081-47bc-9f53-25559992ea53-kube-api-access-klbxx\" (UniqueName: \"kubernetes.io/projected/635f4eda-d081-47bc-9f53-25559992ea53-kube-api-access-klbxx\") pod \"635f4eda-d081-47bc-9f53-25559992ea53\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") "
Nov 03 19:18:22.935298 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935284    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-tmp-dir\") pod \"9c2859a5-32ed-4267-8838-3430edc17c3c\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") "
Nov 03 19:18:22.935433 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935305    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-tmp-dir\") pod \"635f4eda-d081-47bc-9f53-25559992ea53\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") "
Nov 03 19:18:22.935433 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935379    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-model-cache" pod "9c2859a5-32ed-4267-8838-3430edc17c3c" (UID: "9c2859a5-32ed-4267-8838-3430edc17c3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.935433 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935408    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-home\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-home\") pod \"635f4eda-d081-47bc-9f53-25559992ea53\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") "
Nov 03 19:18:22.935581 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935441    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-model-cache\") pod \"635f4eda-d081-47bc-9f53-25559992ea53\" (UID: \"635f4eda-d081-47bc-9f53-25559992ea53\") "
Nov 03 19:18:22.935581 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935520    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2859a5-32ed-4267-8838-3430edc17c3c-kube-api-access-qj5dw\" (UniqueName: \"kubernetes.io/projected/9c2859a5-32ed-4267-8838-3430edc17c3c-kube-api-access-qj5dw\") pod \"9c2859a5-32ed-4267-8838-3430edc17c3c\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") "
Nov 03 19:18:22.935685 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935602    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-tmp-dir" pod "9c2859a5-32ed-4267-8838-3430edc17c3c" (UID: "9c2859a5-32ed-4267-8838-3430edc17c3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.935685 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935608    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-home\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-home\") pod \"9c2859a5-32ed-4267-8838-3430edc17c3c\" (UID: \"9c2859a5-32ed-4267-8838-3430edc17c3c\") "
Nov 03 19:18:22.935844 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935802    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-home" pod "9c2859a5-32ed-4267-8838-3430edc17c3c" (UID: "9c2859a5-32ed-4267-8838-3430edc17c3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.935904 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.935866    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-home" pod "635f4eda-d081-47bc-9f53-25559992ea53" (UID: "635f4eda-d081-47bc-9f53-25559992ea53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.936130 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936040    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-tmp-dir" pod "635f4eda-d081-47bc-9f53-25559992ea53" (UID: "635f4eda-d081-47bc-9f53-25559992ea53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.936235 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936200    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-model-cache" pod "635f4eda-d081-47bc-9f53-25559992ea53" (UID: "635f4eda-d081-47bc-9f53-25559992ea53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.936808 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936781    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:22.936910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936814    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:22.936910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936829    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:22.936910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936843    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:22.936910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936856    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:22.936910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.936869    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:22.937942 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.937908    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2859a5-32ed-4267-8838-3430edc17c3c-tls-certs" pod "9c2859a5-32ed-4267-8838-3430edc17c3c" (UID: "9c2859a5-32ed-4267-8838-3430edc17c3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:22.938051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.937965    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635f4eda-d081-47bc-9f53-25559992ea53-kube-api-access-klbxx" pod "635f4eda-d081-47bc-9f53-25559992ea53" (UID: "635f4eda-d081-47bc-9f53-25559992ea53"). InnerVolumeSpecName "kube-api-access-klbxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:22.938051 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.938028    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-dshm" pod "635f4eda-d081-47bc-9f53-25559992ea53" (UID: "635f4eda-d081-47bc-9f53-25559992ea53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.938155 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.938066    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-dshm" pod "9c2859a5-32ed-4267-8838-3430edc17c3c" (UID: "9c2859a5-32ed-4267-8838-3430edc17c3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:22.938155 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.938123    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635f4eda-d081-47bc-9f53-25559992ea53-tls-certs" pod "635f4eda-d081-47bc-9f53-25559992ea53" (UID: "635f4eda-d081-47bc-9f53-25559992ea53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:22.938314 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:22.938291    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2859a5-32ed-4267-8838-3430edc17c3c-kube-api-access-qj5dw" pod "9c2859a5-32ed-4267-8838-3430edc17c3c" (UID: "9c2859a5-32ed-4267-8838-3430edc17c3c"). InnerVolumeSpecName "kube-api-access-qj5dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:23.013807 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.013773    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d52a40e-0e1d-48ef-91c2-bbf189492e1b" path="/var/lib/kubelet/pods/2d52a40e-0e1d-48ef-91c2-bbf189492e1b/volumes"
Nov 03 19:18:23.014328 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.014306    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" path="/var/lib/kubelet/pods/3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61/volumes"
Nov 03 19:18:23.018914 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.018883    4489 generic.go:356] "Generic (PLEG): container finished" podID="635f4eda-d081-47bc-9f53-25559992ea53" containerID="06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a" exitCode=0
Nov 03 19:18:23.019081 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.019055    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" event={"ID":"635f4eda-d081-47bc-9f53-25559992ea53","Type":"ContainerDied","Data":"06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a"}
Nov 03 19:18:23.019194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.019099    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s" event={"ID":"635f4eda-d081-47bc-9f53-25559992ea53","Type":"ContainerDied","Data":"7e207f42b2b042aa01d1f5429e98d3f40d8d04d6bea488a0af9a58d675fc6054"}
Nov 03 19:18:23.019194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.019119    4489 scope.go:122] "RemoveContainer" containerID="06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a"
Nov 03 19:18:23.019306 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.019068    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"
Nov 03 19:18:23.023620 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.023594    4489 generic.go:356] "Generic (PLEG): container finished" podID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerID="d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7" exitCode=0
Nov 03 19:18:23.023704 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.023638    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" event={"ID":"9c2859a5-32ed-4267-8838-3430edc17c3c","Type":"ContainerDied","Data":"d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7"}
Nov 03 19:18:23.023704 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.023662    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr" event={"ID":"9c2859a5-32ed-4267-8838-3430edc17c3c","Type":"ContainerDied","Data":"257cee8de4152d35e7be731e8619b402c044e22177a2c6fa626e7ca2eef126d6"}
Nov 03 19:18:23.023821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.023727    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"
Nov 03 19:18:23.041358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.039236    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635f4eda-d081-47bc-9f53-25559992ea53-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:23.041358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.039264    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2859a5-32ed-4267-8838-3430edc17c3c-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:23.041358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.039277    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635f4eda-d081-47bc-9f53-25559992ea53-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:23.041358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.039289    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2859a5-32ed-4267-8838-3430edc17c3c-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:23.041358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.039303    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klbxx\" (UniqueName: \"kubernetes.io/projected/635f4eda-d081-47bc-9f53-25559992ea53-kube-api-access-klbxx\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:23.041358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.039316    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5dw\" (UniqueName: \"kubernetes.io/projected/9c2859a5-32ed-4267-8838-3430edc17c3c-kube-api-access-qj5dw\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:23.047713 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.047689    4489 scope.go:122] "RemoveContainer" containerID="06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a"
Nov 03 19:18:23.048063 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:23.048033    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a\": container with ID starting with 06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a not found: ID does not exist" containerID="06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a"
Nov 03 19:18:23.048170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.048072    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a"} err="failed to get container status \"06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a\": rpc error: code = NotFound desc = could not find container \"06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a\": container with ID starting with 06db9f396cbf2a3e59fc352260039d159eac3043f0fe2fe36537a90a85d60f1a not found: ID does not exist"
Nov 03 19:18:23.048170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.048094    4489 scope.go:122] "RemoveContainer" containerID="d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7"
Nov 03 19:18:23.061882 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.061843    4489 scope.go:122] "RemoveContainer" containerID="d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7"
Nov 03 19:18:23.062246 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:23.062210    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7\": container with ID starting with d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7 not found: ID does not exist" containerID="d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7"
Nov 03 19:18:23.062589 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.062560    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7"} err="failed to get container status \"d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7\": rpc error: code = NotFound desc = could not find container \"d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7\": container with ID starting with d1817578dc0d8b16d12757ff05e82fef20d8e546c0722d6249c30d0c860c1ff7 not found: ID does not exist"
Nov 03 19:18:23.329514 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.329434    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"]
Nov 03 19:18:23.335160 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.335119    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/distinct-2-w5-3a7056-kserve-8544ffc9cf-q9gvr"]
Nov 03 19:18:23.454122 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.454094    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"]
Nov 03 19:18:23.457939 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:23.457917    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/premium-w5-3a7056-kserve-7db767c5c7-b797s"]
Nov 03 19:18:24.994642 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:24.994608    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635f4eda-d081-47bc-9f53-25559992ea53" path="/var/lib/kubelet/pods/635f4eda-d081-47bc-9f53-25559992ea53/volumes"
Nov 03 19:18:24.995050 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:24.995036    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" path="/var/lib/kubelet/pods/9c2859a5-32ed-4267-8838-3430edc17c3c/volumes"
Nov 03 19:18:25.060037 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:25.060007    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" podUID="1bec70e8-d473-4799-ac6d-79fa07e44c99" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:18:26.254776 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:26.254733    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"]
Nov 03 19:18:26.277959 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:26.277932    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"]
Nov 03 19:18:26.304829 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:26.304799    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"]
Nov 03 19:18:31.837885 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:31.837851    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"]
Nov 03 19:18:33.363624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:33.363590    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"]
Nov 03 19:18:35.060561 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:35.060514    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" podUID="1bec70e8-d473-4799-ac6d-79fa07e44c99" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:18:38.364780 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:38.364620    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" podUID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" containerName="payload-processing" containerID="cri-o://0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab" gracePeriod=25
Nov 03 19:18:39.940494 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:39.940452    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 19:18:40.021168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.020512    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bec70e8-d473-4799-ac6d-79fa07e44c99-kube-api-access-f9ctd\" (UniqueName: \"kubernetes.io/projected/1bec70e8-d473-4799-ac6d-79fa07e44c99-kube-api-access-f9ctd\") pod \"1bec70e8-d473-4799-ac6d-79fa07e44c99\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") "
Nov 03 19:18:40.021168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.020827    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bec70e8-d473-4799-ac6d-79fa07e44c99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec70e8-d473-4799-ac6d-79fa07e44c99-plugins-config-volume\") pod \"1bec70e8-d473-4799-ac6d-79fa07e44c99\" (UID: \"1bec70e8-d473-4799-ac6d-79fa07e44c99\") "
Nov 03 19:18:40.025167 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.021884    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bec70e8-d473-4799-ac6d-79fa07e44c99-plugins-config-volume" pod "1bec70e8-d473-4799-ac6d-79fa07e44c99" (UID: "1bec70e8-d473-4799-ac6d-79fa07e44c99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:40.033170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.031652    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bec70e8-d473-4799-ac6d-79fa07e44c99-kube-api-access-f9ctd" pod "1bec70e8-d473-4799-ac6d-79fa07e44c99" (UID: "1bec70e8-d473-4799-ac6d-79fa07e44c99"). InnerVolumeSpecName "kube-api-access-f9ctd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:40.131119 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.131029    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ctd\" (UniqueName: \"kubernetes.io/projected/1bec70e8-d473-4799-ac6d-79fa07e44c99-kube-api-access-f9ctd\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:40.132230 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.132209    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bec70e8-d473-4799-ac6d-79fa07e44c99-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:40.188163 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.174550    4489 generic.go:356] "Generic (PLEG): container finished" podID="1bec70e8-d473-4799-ac6d-79fa07e44c99" containerID="a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b" exitCode=0
Nov 03 19:18:40.188163 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.174706    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" event={"ID":"1bec70e8-d473-4799-ac6d-79fa07e44c99","Type":"ContainerDied","Data":"a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b"}
Nov 03 19:18:40.188163 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.174781    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m" event={"ID":"1bec70e8-d473-4799-ac6d-79fa07e44c99","Type":"ContainerDied","Data":"9ba5dc969985b9d4239026b7d1fe9cc9e06b2d9c95dc507bbfb43771c33ef4c8"}
Nov 03 19:18:40.188163 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.174832    4489 scope.go:122] "RemoveContainer" containerID="a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b"
Nov 03 19:18:40.188163 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.175026    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"
Nov 03 19:18:40.218678 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.217646    4489 scope.go:122] "RemoveContainer" containerID="a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b"
Nov 03 19:18:40.219026 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:40.218982    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b\": container with ID starting with a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b not found: ID does not exist" containerID="a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b"
Nov 03 19:18:40.219199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.219172    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b"} err="failed to get container status \"a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b\": rpc error: code = NotFound desc = could not find container \"a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b\": container with ID starting with a406ede984326f440ffa4e53aa2ff1ddd09f41290ccc4dd2c204abc2d5708f8b not found: ID does not exist"
Nov 03 19:18:40.349753 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.349714    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"]
Nov 03 19:18:40.354564 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:40.354537    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3a7056-68d95fb54c-8kq4m"]
Nov 03 19:18:41.014220 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.009268    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bec70e8-d473-4799-ac6d-79fa07e44c99" path="/var/lib/kubelet/pods/1bec70e8-d473-4799-ac6d-79fa07e44c99/volumes"
Nov 03 19:18:41.275732 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.275636    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main" containerID="cri-o://f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8" gracePeriod=45
Nov 03 19:18:41.297757 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.297719    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" containerID="cri-o://de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458" gracePeriod=45
Nov 03 19:18:41.324154 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.324103    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main" containerID="cri-o://995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26" gracePeriod=45
Nov 03 19:18:41.517685 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.517630    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Nov 03 19:18:41.669162 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.667950    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 19:18:41.671389 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.671362    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:18:41.675094 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.675072    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 19:18:41.678501 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678473    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tmp-dir\") pod \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678532    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tls-certs\") pod \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678572    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-model-cache\") pod \"ddd329e8-de84-4261-b3fe-8e17b51c550a\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678603    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd329e8-de84-4261-b3fe-8e17b51c550a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd329e8-de84-4261-b3fe-8e17b51c550a-tls-certs\") pod \"ddd329e8-de84-4261-b3fe-8e17b51c550a\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678643    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-tmp-dir\") pod \"ddd329e8-de84-4261-b3fe-8e17b51c550a\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678730    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-dshm\") pod \"ddd329e8-de84-4261-b3fe-8e17b51c550a\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678749    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-model-cache\") pod \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678750    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tmp-dir" pod "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" (UID: "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678793    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-home\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-home\") pod \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678837    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-kube-api-access-zc8f9\" (UniqueName: \"kubernetes.io/projected/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-kube-api-access-zc8f9\") pod \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678866    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd329e8-de84-4261-b3fe-8e17b51c550a-kube-api-access-pg9xz\" (UniqueName: \"kubernetes.io/projected/ddd329e8-de84-4261-b3fe-8e17b51c550a-kube-api-access-pg9xz\") pod \"ddd329e8-de84-4261-b3fe-8e17b51c550a\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678902    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-dshm\") pod \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\" (UID: \"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7\") "
Nov 03 19:18:41.679063 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.678934    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-home\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-home\") pod \"ddd329e8-de84-4261-b3fe-8e17b51c550a\" (UID: \"ddd329e8-de84-4261-b3fe-8e17b51c550a\") "
Nov 03 19:18:41.679618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.679305    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.679618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.679314    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-model-cache" pod "ddd329e8-de84-4261-b3fe-8e17b51c550a" (UID: "ddd329e8-de84-4261-b3fe-8e17b51c550a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.679618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.679516    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-home" pod "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" (UID: "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.679618 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.679523    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-home" pod "ddd329e8-de84-4261-b3fe-8e17b51c550a" (UID: "ddd329e8-de84-4261-b3fe-8e17b51c550a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.679793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.679714    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-tmp-dir" pod "ddd329e8-de84-4261-b3fe-8e17b51c550a" (UID: "ddd329e8-de84-4261-b3fe-8e17b51c550a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.679878 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.679837    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-model-cache" pod "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" (UID: "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.683349 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.683309    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-kube-api-access-zc8f9" pod "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" (UID: "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7"). InnerVolumeSpecName "kube-api-access-zc8f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:41.683504 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.683476    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd329e8-de84-4261-b3fe-8e17b51c550a-tls-certs" pod "ddd329e8-de84-4261-b3fe-8e17b51c550a" (UID: "ddd329e8-de84-4261-b3fe-8e17b51c550a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:41.683596 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.683572    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd329e8-de84-4261-b3fe-8e17b51c550a-kube-api-access-pg9xz" pod "ddd329e8-de84-4261-b3fe-8e17b51c550a" (UID: "ddd329e8-de84-4261-b3fe-8e17b51c550a"). InnerVolumeSpecName "kube-api-access-pg9xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:41.685131 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.685101    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tls-certs" pod "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" (UID: "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:41.685683 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.685654    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-dshm" pod "ddd329e8-de84-4261-b3fe-8e17b51c550a" (UID: "ddd329e8-de84-4261-b3fe-8e17b51c550a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.689325 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.689290    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-dshm" pod "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" (UID: "5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.780677 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.780645    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-dshm\") pod \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") "
Nov 03 19:18:41.780852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.780756    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-home\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-home\") pod \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") "
Nov 03 19:18:41.780852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.780794    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ea7133-1b1f-44ab-a003-ff63a40a72f9-kube-api-access-jmqgv\" (UniqueName: \"kubernetes.io/projected/74ea7133-1b1f-44ab-a003-ff63a40a72f9-kube-api-access-jmqgv\") pod \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") "
Nov 03 19:18:41.780941 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.780862    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tmp-dir\") pod \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") "
Nov 03 19:18:41.780980 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.780952    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tls-certs\") pod \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") "
Nov 03 19:18:41.781020 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.780974    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-model-cache\") pod \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\" (UID: \"74ea7133-1b1f-44ab-a003-ff63a40a72f9\") "
Nov 03 19:18:41.781020 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.780998    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-home" pod "74ea7133-1b1f-44ab-a003-ff63a40a72f9" (UID: "74ea7133-1b1f-44ab-a003-ff63a40a72f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.781288 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781258    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tmp-dir" pod "74ea7133-1b1f-44ab-a003-ff63a40a72f9" (UID: "74ea7133-1b1f-44ab-a003-ff63a40a72f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781369    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-model-cache" pod "74ea7133-1b1f-44ab-a003-ff63a40a72f9" (UID: "74ea7133-1b1f-44ab-a003-ff63a40a72f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781440    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781580    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8f9\" (UniqueName: \"kubernetes.io/projected/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-kube-api-access-zc8f9\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781598    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg9xz\" (UniqueName: \"kubernetes.io/projected/ddd329e8-de84-4261-b3fe-8e17b51c550a-kube-api-access-pg9xz\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781612    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781626    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781638    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781651    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781665    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781680    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd329e8-de84-4261-b3fe-8e17b51c550a-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781693    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781710    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781723    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd329e8-de84-4261-b3fe-8e17b51c550a-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.782248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.781736    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.785390 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.785250    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ea7133-1b1f-44ab-a003-ff63a40a72f9-kube-api-access-jmqgv" pod "74ea7133-1b1f-44ab-a003-ff63a40a72f9" (UID: "74ea7133-1b1f-44ab-a003-ff63a40a72f9"). InnerVolumeSpecName "kube-api-access-jmqgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:41.785390 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.785353    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-dshm" pod "74ea7133-1b1f-44ab-a003-ff63a40a72f9" (UID: "74ea7133-1b1f-44ab-a003-ff63a40a72f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:41.787688 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.787665    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tls-certs" pod "74ea7133-1b1f-44ab-a003-ff63a40a72f9" (UID: "74ea7133-1b1f-44ab-a003-ff63a40a72f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:41.883294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.883222    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmqgv\" (UniqueName: \"kubernetes.io/projected/74ea7133-1b1f-44ab-a003-ff63a40a72f9-kube-api-access-jmqgv\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.883294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.883248    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74ea7133-1b1f-44ab-a003-ff63a40a72f9-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.883294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.883257    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:41.883294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:41.883267    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ea7133-1b1f-44ab-a003-ff63a40a72f9-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:42.190953 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.190920    4489 generic.go:356] "Generic (PLEG): container finished" podID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerID="de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458" exitCode=0
Nov 03 19:18:42.191464 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.190975    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"
Nov 03 19:18:42.191464 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.190995    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" event={"ID":"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7","Type":"ContainerDied","Data":"de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458"}
Nov 03 19:18:42.191464 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.191037    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g" event={"ID":"5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7","Type":"ContainerDied","Data":"5a594b534ef9775ce2113856ebfcc74f2db592c054dc4d2285d6348721ac0934"}
Nov 03 19:18:42.191464 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.191054    4489 scope.go:122] "RemoveContainer" containerID="de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458"
Nov 03 19:18:42.203006 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.195198    4489 generic.go:356] "Generic (PLEG): container finished" podID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerID="f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8" exitCode=0
Nov 03 19:18:42.203006 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.195336    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"
Nov 03 19:18:42.203006 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.195688    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" event={"ID":"ddd329e8-de84-4261-b3fe-8e17b51c550a","Type":"ContainerDied","Data":"f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8"}
Nov 03 19:18:42.203006 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.195729    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f" event={"ID":"ddd329e8-de84-4261-b3fe-8e17b51c550a","Type":"ContainerDied","Data":"f60f7baa136e1418354808387ac46d9c79a202c429c87a2e3e9922fbed42a20c"}
Nov 03 19:18:42.215185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.207954    4489 scope.go:122] "RemoveContainer" containerID="de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458"
Nov 03 19:18:42.215185 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:42.208400    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458\": container with ID starting with de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458 not found: ID does not exist" containerID="de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458"
Nov 03 19:18:42.215185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.208431    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458"} err="failed to get container status \"de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458\": rpc error: code = NotFound desc = could not find container \"de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458\": container with ID starting with de47b5db82b178eb0ab08b8f05d37f949f9d767cca2f04004f1ead951fdd7458 not found: ID does not exist"
Nov 03 19:18:42.215185 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.208454    4489 scope.go:122] "RemoveContainer" containerID="f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8"
Nov 03 19:18:42.217490 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.217465    4489 generic.go:356] "Generic (PLEG): container finished" podID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerID="995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26" exitCode=0
Nov 03 19:18:42.217600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.217545    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"
Nov 03 19:18:42.217600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.217546    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" event={"ID":"74ea7133-1b1f-44ab-a003-ff63a40a72f9","Type":"ContainerDied","Data":"995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26"}
Nov 03 19:18:42.217600 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.217595    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49" event={"ID":"74ea7133-1b1f-44ab-a003-ff63a40a72f9","Type":"ContainerDied","Data":"f151a1e465002f4a1bee0670e6efe3babb8c9fd02f719c81cbec8143d5b6dd43"}
Nov 03 19:18:42.221940 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.221923    4489 scope.go:122] "RemoveContainer" containerID="f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8"
Nov 03 19:18:42.222346 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:42.222317    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8\": container with ID starting with f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8 not found: ID does not exist" containerID="f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8"
Nov 03 19:18:42.222471 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.222355    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8"} err="failed to get container status \"f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8\": rpc error: code = NotFound desc = could not find container \"f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8\": container with ID starting with f9eebb80349f7b8d52ea409339304c9e7d7654858014176f3fea9bc75a0281c8 not found: ID does not exist"
Nov 03 19:18:42.222471 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.222374    4489 scope.go:122] "RemoveContainer" containerID="995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26"
Nov 03 19:18:42.232681 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.232667    4489 scope.go:122] "RemoveContainer" containerID="995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26"
Nov 03 19:18:42.232983 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:42.232966    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26\": container with ID starting with 995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26 not found: ID does not exist" containerID="995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26"
Nov 03 19:18:42.233054 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.232990    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26"} err="failed to get container status \"995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26\": rpc error: code = NotFound desc = could not find container \"995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26\": container with ID starting with 995eba0a56b270f38ceb89a165d1faa66f37e7edc57fbcbb0904c4fd4562ad26 not found: ID does not exist"
Nov 03 19:18:42.364200 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.364164    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"]
Nov 03 19:18:42.368468 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.368432    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-w0-9686c3-kserve-5d7f469658-hk24g"]
Nov 03 19:18:42.481938 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.481852    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"]
Nov 03 19:18:42.485401 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.485373    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/simulator-w0-9686c3-kserve-57cbb49666-8tx49"]
Nov 03 19:18:42.601101 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.601064    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"]
Nov 03 19:18:42.604689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.604666    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/distinct-2-w0-9686c3-kserve-689446455c-2sx2f"]
Nov 03 19:18:42.994533 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.994500    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" path="/var/lib/kubelet/pods/5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7/volumes"
Nov 03 19:18:42.994896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.994882    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" path="/var/lib/kubelet/pods/74ea7133-1b1f-44ab-a003-ff63a40a72f9/volumes"
Nov 03 19:18:42.995263 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:42.995250    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" path="/var/lib/kubelet/pods/ddd329e8-de84-4261-b3fe-8e17b51c550a/volumes"
Nov 03 19:18:43.038692 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:43.038651    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" podUID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:18:46.861697 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:46.861638    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main" containerID="cri-o://cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a" gracePeriod=45
Nov 03 19:18:47.124838 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.124771    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:18:47.246644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.246609    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tls-certs\") pod \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") "
Nov 03 19:18:47.246821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.246746    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tmp-dir\") pod \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") "
Nov 03 19:18:47.246821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.246811    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-dshm\") pod \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") "
Nov 03 19:18:47.246961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.246864    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-model-cache\") pod \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") "
Nov 03 19:18:47.246961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.246898    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-kube-api-access-97bnr\" (UniqueName: \"kubernetes.io/projected/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-kube-api-access-97bnr\") pod \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") "
Nov 03 19:18:47.247074 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.246972    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-home\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-home\") pod \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\" (UID: \"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8\") "
Nov 03 19:18:47.247775 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.247741    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-home" pod "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" (UID: "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:47.247882 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.247814    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-model-cache" pod "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" (UID: "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:47.247922 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.247892    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tmp-dir" pod "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" (UID: "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:47.249630 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.249597    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tls-certs" pod "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" (UID: "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:47.250055 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.250015    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-dshm" pod "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" (UID: "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:47.250198 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.250173    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-kube-api-access-97bnr" pod "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" (UID: "ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8"). InnerVolumeSpecName "kube-api-access-97bnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:47.263625 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.263596    4489 generic.go:356] "Generic (PLEG): container finished" podID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerID="cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a" exitCode=0
Nov 03 19:18:47.263776 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.263737    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" event={"ID":"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8","Type":"ContainerDied","Data":"cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a"}
Nov 03 19:18:47.263844 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.263800    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx" event={"ID":"ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8","Type":"ContainerDied","Data":"5a1fe2b1496e59ec779ebbf638849f292f1e8005c5100ca5aeb7f273044fc4e1"}
Nov 03 19:18:47.263844 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.263821    4489 scope.go:122] "RemoveContainer" containerID="cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a"
Nov 03 19:18:47.263974 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.263961    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"
Nov 03 19:18:47.276950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.276921    4489 scope.go:122] "RemoveContainer" containerID="cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a"
Nov 03 19:18:47.277307 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:47.277284    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a\": container with ID starting with cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a not found: ID does not exist" containerID="cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a"
Nov 03 19:18:47.277394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.277314    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a"} err="failed to get container status \"cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a\": rpc error: code = NotFound desc = could not find container \"cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a\": container with ID starting with cc475fb33f6dc1fc9a27a3122ac08de43765db84e82857090921a84218bf877a not found: ID does not exist"
Nov 03 19:18:47.348768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.348734    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:47.348768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.348770    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:47.348948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.348782    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:47.348948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.348798    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97bnr\" (UniqueName: \"kubernetes.io/projected/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-kube-api-access-97bnr\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:47.348948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.348814    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:47.348948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.348827    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:47.440206 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.440124    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"]
Nov 03 19:18:47.443122 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:47.443095    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/simulator-w3-a4cacd-kserve-67885cb984-w2lfx"]
Nov 03 19:18:47.534416 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:47.534375    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:18:48.995188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:48.995135    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" path="/var/lib/kubelet/pods/ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8/volumes"
Nov 03 19:18:53.036254 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:53.036204    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" podUID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:18:58.624987 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:58.624926    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 19:18:58.789583 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:58.789552    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-kube-api-access-fvs5k\" (UniqueName: \"kubernetes.io/projected/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-kube-api-access-fvs5k\") pod \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") "
Nov 03 19:18:58.789757 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:58.789745    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-plugins-config-volume\") pod \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\" (UID: \"7a2e4eaf-8cea-46bb-9327-9bce2f88e081\") "
Nov 03 19:18:58.790098 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:58.790073    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-plugins-config-volume" pod "7a2e4eaf-8cea-46bb-9327-9bce2f88e081" (UID: "7a2e4eaf-8cea-46bb-9327-9bce2f88e081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:58.791746 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:58.791716    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-kube-api-access-fvs5k" pod "7a2e4eaf-8cea-46bb-9327-9bce2f88e081" (UID: "7a2e4eaf-8cea-46bb-9327-9bce2f88e081"). InnerVolumeSpecName "kube-api-access-fvs5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:58.891355 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:58.891295    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:58.891355 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:58.891318    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvs5k\" (UniqueName: \"kubernetes.io/projected/7a2e4eaf-8cea-46bb-9327-9bce2f88e081-kube-api-access-fvs5k\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:59.340101 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.340057    4489 generic.go:356] "Generic (PLEG): container finished" podID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" containerID="0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab" exitCode=0
Nov 03 19:18:59.340297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.340112    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" event={"ID":"7a2e4eaf-8cea-46bb-9327-9bce2f88e081","Type":"ContainerDied","Data":"0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab"}
Nov 03 19:18:59.340297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.340129    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"
Nov 03 19:18:59.340297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.340190    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq" event={"ID":"7a2e4eaf-8cea-46bb-9327-9bce2f88e081","Type":"ContainerDied","Data":"779ca4bd7108b69e0969516ca5df3be0e3338c7d5d01ad300b70f5140ccb64e8"}
Nov 03 19:18:59.340297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.340221    4489 scope.go:122] "RemoveContainer" containerID="0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab"
Nov 03 19:18:59.349164 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.349126    4489 scope.go:122] "RemoveContainer" containerID="0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab"
Nov 03 19:18:59.349425 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:18:59.349403    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab\": container with ID starting with 0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab not found: ID does not exist" containerID="0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab"
Nov 03 19:18:59.349492 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.349433    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab"} err="failed to get container status \"0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab\": rpc error: code = NotFound desc = could not find container \"0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab\": container with ID starting with 0064c3b23bf501dc42bb606f28e6dc4265fb2dbcd1f1f81968e43cd3deeb56ab not found: ID does not exist"
Nov 03 19:18:59.510444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.510409    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"]
Nov 03 19:18:59.562195 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:18:59.562163    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9686c3-779d4f5849-mnqkq"]
Nov 03 19:19:00.995303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:00.995272    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" path="/var/lib/kubelet/pods/7a2e4eaf-8cea-46bb-9327-9bce2f88e081/volumes"
Nov 03 19:19:17.531741 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:19:17.531700    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:19:24.913174 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:24.913124    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"]
Nov 03 19:19:24.913608 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:24.913369    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66" podUID="126f904d-4a17-4e30-95e9-8e343a5f613f" containerName="maas-api" containerID="cri-o://4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a" gracePeriod=30
Nov 03 19:19:25.198110 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.198081    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 19:19:25.318244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.317200    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126f904d-4a17-4e30-95e9-8e343a5f613f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f904d-4a17-4e30-95e9-8e343a5f613f-openshift-service-ca\") pod \"126f904d-4a17-4e30-95e9-8e343a5f613f\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") "
Nov 03 19:19:25.318244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.317253    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126f904d-4a17-4e30-95e9-8e343a5f613f-kube-api-access-67w5v\" (UniqueName: \"kubernetes.io/projected/126f904d-4a17-4e30-95e9-8e343a5f613f-kube-api-access-67w5v\") pod \"126f904d-4a17-4e30-95e9-8e343a5f613f\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") "
Nov 03 19:19:25.318244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.317325    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-metrics-certs\") pod \"126f904d-4a17-4e30-95e9-8e343a5f613f\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") "
Nov 03 19:19:25.318244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.317397    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-maas-api-tls\") pod \"126f904d-4a17-4e30-95e9-8e343a5f613f\" (UID: \"126f904d-4a17-4e30-95e9-8e343a5f613f\") "
Nov 03 19:19:25.320221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.320171    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126f904d-4a17-4e30-95e9-8e343a5f613f-openshift-service-ca" pod "126f904d-4a17-4e30-95e9-8e343a5f613f" (UID: "126f904d-4a17-4e30-95e9-8e343a5f613f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:25.321352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.321326    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126f904d-4a17-4e30-95e9-8e343a5f613f-kube-api-access-67w5v" pod "126f904d-4a17-4e30-95e9-8e343a5f613f" (UID: "126f904d-4a17-4e30-95e9-8e343a5f613f"). InnerVolumeSpecName "kube-api-access-67w5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:25.323529 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.323502    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-metrics-certs" pod "126f904d-4a17-4e30-95e9-8e343a5f613f" (UID: "126f904d-4a17-4e30-95e9-8e343a5f613f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:25.325499 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.325470    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-maas-api-tls" pod "126f904d-4a17-4e30-95e9-8e343a5f613f" (UID: "126f904d-4a17-4e30-95e9-8e343a5f613f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:25.418652 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.418616    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/126f904d-4a17-4e30-95e9-8e343a5f613f-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:25.418652 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.418647    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67w5v\" (UniqueName: \"kubernetes.io/projected/126f904d-4a17-4e30-95e9-8e343a5f613f-kube-api-access-67w5v\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:25.418845 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.418661    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:25.418845 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.418672    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/126f904d-4a17-4e30-95e9-8e343a5f613f-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:25.480545 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.480450    4489 generic.go:356] "Generic (PLEG): container finished" podID="126f904d-4a17-4e30-95e9-8e343a5f613f" containerID="4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a" exitCode=0
Nov 03 19:19:25.480545 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.480501    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"
Nov 03 19:19:25.480545 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.480528    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66" event={"ID":"126f904d-4a17-4e30-95e9-8e343a5f613f","Type":"ContainerDied","Data":"4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a"}
Nov 03 19:19:25.480816 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.480587    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66" event={"ID":"126f904d-4a17-4e30-95e9-8e343a5f613f","Type":"ContainerDied","Data":"7723dcac2010a2e405b8c0108cdb49b1dec5dc68c139cf11e19c6de7adc1a6ef"}
Nov 03 19:19:25.480816 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.480612    4489 scope.go:122] "RemoveContainer" containerID="4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a"
Nov 03 19:19:25.491818 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.491803    4489 scope.go:122] "RemoveContainer" containerID="4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a"
Nov 03 19:19:25.492153 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:19:25.492116    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a\": container with ID starting with 4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a not found: ID does not exist" containerID="4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a"
Nov 03 19:19:25.492207 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.492165    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a"} err="failed to get container status \"4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a\": rpc error: code = NotFound desc = could not find container \"4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a\": container with ID starting with 4dcdd28ddb7ee913742b799db8cf208fb2f330380e085b68435a631a01425e8a not found: ID does not exist"
Nov 03 19:19:25.650585 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.650559    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"]
Nov 03 19:19:25.654010 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:25.653989    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3a7056-8cb98f5d4-klp66"]
Nov 03 19:19:26.994992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:26.994953    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126f904d-4a17-4e30-95e9-8e343a5f613f" path="/var/lib/kubelet/pods/126f904d-4a17-4e30-95e9-8e343a5f613f/volumes"
Nov 03 19:19:35.527319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.527284    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"]
Nov 03 19:19:35.527840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.527521    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9" podUID="5707bcc4-3660-4c82-9bf5-c2858973132f" containerName="maas-api" containerID="cri-o://4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb" gracePeriod=30
Nov 03 19:19:35.817047 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.817020    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:19:35.878836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.878804    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5707bcc4-3660-4c82-9bf5-c2858973132f-kube-api-access-rxsxc\" (UniqueName: \"kubernetes.io/projected/5707bcc4-3660-4c82-9bf5-c2858973132f-kube-api-access-rxsxc\") pod \"5707bcc4-3660-4c82-9bf5-c2858973132f\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") "
Nov 03 19:19:35.878997 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.878930    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-metrics-certs\") pod \"5707bcc4-3660-4c82-9bf5-c2858973132f\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") "
Nov 03 19:19:35.879039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.879018    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5707bcc4-3660-4c82-9bf5-c2858973132f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5707bcc4-3660-4c82-9bf5-c2858973132f-openshift-service-ca\") pod \"5707bcc4-3660-4c82-9bf5-c2858973132f\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") "
Nov 03 19:19:35.879078 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.879039    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls\") pod \"5707bcc4-3660-4c82-9bf5-c2858973132f\" (UID: \"5707bcc4-3660-4c82-9bf5-c2858973132f\") "
Nov 03 19:19:35.879483 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.879453    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5707bcc4-3660-4c82-9bf5-c2858973132f-openshift-service-ca" pod "5707bcc4-3660-4c82-9bf5-c2858973132f" (UID: "5707bcc4-3660-4c82-9bf5-c2858973132f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:35.881177 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.881132    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5707bcc4-3660-4c82-9bf5-c2858973132f-kube-api-access-rxsxc" pod "5707bcc4-3660-4c82-9bf5-c2858973132f" (UID: "5707bcc4-3660-4c82-9bf5-c2858973132f"). InnerVolumeSpecName "kube-api-access-rxsxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:35.881261 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.881174    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-metrics-certs" pod "5707bcc4-3660-4c82-9bf5-c2858973132f" (UID: "5707bcc4-3660-4c82-9bf5-c2858973132f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:35.881404 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.881382    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls" pod "5707bcc4-3660-4c82-9bf5-c2858973132f" (UID: "5707bcc4-3660-4c82-9bf5-c2858973132f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:35.980082 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.980049    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxsxc\" (UniqueName: \"kubernetes.io/projected/5707bcc4-3660-4c82-9bf5-c2858973132f-kube-api-access-rxsxc\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:35.980082 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.980078    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:35.980082 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.980088    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5707bcc4-3660-4c82-9bf5-c2858973132f-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:35.980334 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:35.980096    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5707bcc4-3660-4c82-9bf5-c2858973132f-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:36.545414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.545382    4489 generic.go:356] "Generic (PLEG): container finished" podID="5707bcc4-3660-4c82-9bf5-c2858973132f" containerID="4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb" exitCode=0
Nov 03 19:19:36.545836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.545421    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"
Nov 03 19:19:36.545836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.545464    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9" event={"ID":"5707bcc4-3660-4c82-9bf5-c2858973132f","Type":"ContainerDied","Data":"4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb"}
Nov 03 19:19:36.545836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.545521    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9" event={"ID":"5707bcc4-3660-4c82-9bf5-c2858973132f","Type":"ContainerDied","Data":"f806e613f6c3e63fb6e93426327cac397d2cb6e18d8ef0c4ce4d9cd024b33857"}
Nov 03 19:19:36.545836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.545549    4489 scope.go:122] "RemoveContainer" containerID="4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb"
Nov 03 19:19:36.555134 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.555115    4489 scope.go:122] "RemoveContainer" containerID="4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb"
Nov 03 19:19:36.555431 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:19:36.555412    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb\": container with ID starting with 4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb not found: ID does not exist" containerID="4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb"
Nov 03 19:19:36.555493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.555438    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb"} err="failed to get container status \"4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb\": rpc error: code = NotFound desc = could not find container \"4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb\": container with ID starting with 4ce482248c39267426660594c78ec4ca7141b15ce519765d86fc77841eb53dfb not found: ID does not exist"
Nov 03 19:19:36.715907 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.715878    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"]
Nov 03 19:19:36.767666 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.767628    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9e277a-5c7cd97697-qv8g9"]
Nov 03 19:19:36.994832 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:36.994795    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5707bcc4-3660-4c82-9bf5-c2858973132f" path="/var/lib/kubelet/pods/5707bcc4-3660-4c82-9bf5-c2858973132f/volumes"
Nov 03 19:19:41.514411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.514376    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"]
Nov 03 19:19:41.514822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.514585    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt" podUID="fe7755b4-3a94-46eb-8ff4-3c651c11c230" containerName="maas-api" containerID="cri-o://2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e" gracePeriod=30
Nov 03 19:19:41.791192 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.791166    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 19:19:41.850823 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.850790    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7755b4-3a94-46eb-8ff4-3c651c11c230-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7755b4-3a94-46eb-8ff4-3c651c11c230-openshift-service-ca\") pod \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") "
Nov 03 19:19:41.850973 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.850872    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7755b4-3a94-46eb-8ff4-3c651c11c230-kube-api-access-bzrqp\" (UniqueName: \"kubernetes.io/projected/fe7755b4-3a94-46eb-8ff4-3c651c11c230-kube-api-access-bzrqp\") pod \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") "
Nov 03 19:19:41.850973 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.850932    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls\") pod \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") "
Nov 03 19:19:41.851100 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.850978    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-metrics-certs\") pod \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\" (UID: \"fe7755b4-3a94-46eb-8ff4-3c651c11c230\") "
Nov 03 19:19:41.851304 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.851278    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7755b4-3a94-46eb-8ff4-3c651c11c230-openshift-service-ca" pod "fe7755b4-3a94-46eb-8ff4-3c651c11c230" (UID: "fe7755b4-3a94-46eb-8ff4-3c651c11c230"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:41.851549 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.851532    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7755b4-3a94-46eb-8ff4-3c651c11c230-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:41.853375 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.853348    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls" pod "fe7755b4-3a94-46eb-8ff4-3c651c11c230" (UID: "fe7755b4-3a94-46eb-8ff4-3c651c11c230"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:41.853465 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.853404    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7755b4-3a94-46eb-8ff4-3c651c11c230-kube-api-access-bzrqp" pod "fe7755b4-3a94-46eb-8ff4-3c651c11c230" (UID: "fe7755b4-3a94-46eb-8ff4-3c651c11c230"). InnerVolumeSpecName "kube-api-access-bzrqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:41.853465 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.853438    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-metrics-certs" pod "fe7755b4-3a94-46eb-8ff4-3c651c11c230" (UID: "fe7755b4-3a94-46eb-8ff4-3c651c11c230"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:41.952616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.952474    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrqp\" (UniqueName: \"kubernetes.io/projected/fe7755b4-3a94-46eb-8ff4-3c651c11c230-kube-api-access-bzrqp\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:41.952616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.952511    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:41.952616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:41.952526    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7755b4-3a94-46eb-8ff4-3c651c11c230-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:42.584806 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.584774    4489 generic.go:356] "Generic (PLEG): container finished" podID="fe7755b4-3a94-46eb-8ff4-3c651c11c230" containerID="2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e" exitCode=0
Nov 03 19:19:42.585243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.584820    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"
Nov 03 19:19:42.585243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.584858    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt" event={"ID":"fe7755b4-3a94-46eb-8ff4-3c651c11c230","Type":"ContainerDied","Data":"2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e"}
Nov 03 19:19:42.585243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.584921    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt" event={"ID":"fe7755b4-3a94-46eb-8ff4-3c651c11c230","Type":"ContainerDied","Data":"331c76a24d3bb6271e3b2b90c20bf32edf54e26b730b67af588cd886338f3e1e"}
Nov 03 19:19:42.585243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.584951    4489 scope.go:122] "RemoveContainer" containerID="2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e"
Nov 03 19:19:42.594911 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.594890    4489 scope.go:122] "RemoveContainer" containerID="2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e"
Nov 03 19:19:42.595186 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:19:42.595166    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e\": container with ID starting with 2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e not found: ID does not exist" containerID="2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e"
Nov 03 19:19:42.595272 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.595196    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e"} err="failed to get container status \"2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e\": rpc error: code = NotFound desc = could not find container \"2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e\": container with ID starting with 2312d4c07885b8f7fc3d7bdb32ddce07742d348ae6ac263515d9183db49e801e not found: ID does not exist"
Nov 03 19:19:42.755909 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.755878    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"]
Nov 03 19:19:42.808232 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.808192    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9686c3-7b646f6498-4x4kt"]
Nov 03 19:19:42.999327 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:42.997513    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7755b4-3a94-46eb-8ff4-3c651c11c230" path="/var/lib/kubelet/pods/fe7755b4-3a94-46eb-8ff4-3c651c11c230/volumes"
Nov 03 19:19:47.530073 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:19:47.530039    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:19:57.183942 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:57.183906    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"]
Nov 03 19:19:57.299774 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:19:57.299742    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"]
Nov 03 19:20:02.184777 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.184716    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9" podUID="c73eaba1-ec4e-4348-a45c-b364300a95a3" containerName="payload-processing" containerID="cri-o://98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d" gracePeriod=25
Nov 03 19:20:02.300378 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.300332    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78" podUID="fbe49d37-6d34-49a2-8348-8dad98fd6883" containerName="payload-pre-processing" containerID="cri-o://3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2" gracePeriod=25
Nov 03 19:20:02.448607 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.448537    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:20:02.531507 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.526577    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73eaba1-ec4e-4348-a45c-b364300a95a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73eaba1-ec4e-4348-a45c-b364300a95a3-plugins-config-volume\") pod \"c73eaba1-ec4e-4348-a45c-b364300a95a3\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") "
Nov 03 19:20:02.531507 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.526712    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73eaba1-ec4e-4348-a45c-b364300a95a3-kube-api-access-vtjdj\" (UniqueName: \"kubernetes.io/projected/c73eaba1-ec4e-4348-a45c-b364300a95a3-kube-api-access-vtjdj\") pod \"c73eaba1-ec4e-4348-a45c-b364300a95a3\" (UID: \"c73eaba1-ec4e-4348-a45c-b364300a95a3\") "
Nov 03 19:20:02.532624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.532562    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73eaba1-ec4e-4348-a45c-b364300a95a3-plugins-config-volume" pod "c73eaba1-ec4e-4348-a45c-b364300a95a3" (UID: "c73eaba1-ec4e-4348-a45c-b364300a95a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:02.534078 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.534019    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73eaba1-ec4e-4348-a45c-b364300a95a3-kube-api-access-vtjdj" pod "c73eaba1-ec4e-4348-a45c-b364300a95a3" (UID: "c73eaba1-ec4e-4348-a45c-b364300a95a3"). InnerVolumeSpecName "kube-api-access-vtjdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:02.577221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.577191    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:20:02.628043 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.628008    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe49d37-6d34-49a2-8348-8dad98fd6883-kube-api-access-cfzjx\" (UniqueName: \"kubernetes.io/projected/fbe49d37-6d34-49a2-8348-8dad98fd6883-kube-api-access-cfzjx\") pod \"fbe49d37-6d34-49a2-8348-8dad98fd6883\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") "
Nov 03 19:20:02.628266 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.628087    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe49d37-6d34-49a2-8348-8dad98fd6883-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe49d37-6d34-49a2-8348-8dad98fd6883-plugins-config-volume\") pod \"fbe49d37-6d34-49a2-8348-8dad98fd6883\" (UID: \"fbe49d37-6d34-49a2-8348-8dad98fd6883\") "
Nov 03 19:20:02.628340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.628319    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtjdj\" (UniqueName: \"kubernetes.io/projected/c73eaba1-ec4e-4348-a45c-b364300a95a3-kube-api-access-vtjdj\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:02.628340 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.628332    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73eaba1-ec4e-4348-a45c-b364300a95a3-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:02.628545 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.628519    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe49d37-6d34-49a2-8348-8dad98fd6883-plugins-config-volume" pod "fbe49d37-6d34-49a2-8348-8dad98fd6883" (UID: "fbe49d37-6d34-49a2-8348-8dad98fd6883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:02.630194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.630176    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe49d37-6d34-49a2-8348-8dad98fd6883-kube-api-access-cfzjx" pod "fbe49d37-6d34-49a2-8348-8dad98fd6883" (UID: "fbe49d37-6d34-49a2-8348-8dad98fd6883"). InnerVolumeSpecName "kube-api-access-cfzjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:02.687039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.687002    4489 generic.go:356] "Generic (PLEG): container finished" podID="c73eaba1-ec4e-4348-a45c-b364300a95a3" containerID="98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d" exitCode=0
Nov 03 19:20:02.687203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.687066    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"
Nov 03 19:20:02.687203 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.687081    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9" event={"ID":"c73eaba1-ec4e-4348-a45c-b364300a95a3","Type":"ContainerDied","Data":"98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d"}
Nov 03 19:20:02.687323 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.687216    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9" event={"ID":"c73eaba1-ec4e-4348-a45c-b364300a95a3","Type":"ContainerDied","Data":"57c1d399969cadefbd53e47f83d607ecfe9cefdde669c59e3f8586f9933d0fad"}
Nov 03 19:20:02.687323 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.687253    4489 scope.go:122] "RemoveContainer" containerID="98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d"
Nov 03 19:20:02.688519 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.688500    4489 generic.go:356] "Generic (PLEG): container finished" podID="fbe49d37-6d34-49a2-8348-8dad98fd6883" containerID="3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2" exitCode=0
Nov 03 19:20:02.688615 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.688568    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"
Nov 03 19:20:02.688615 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.688580    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78" event={"ID":"fbe49d37-6d34-49a2-8348-8dad98fd6883","Type":"ContainerDied","Data":"3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2"}
Nov 03 19:20:02.688615 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.688614    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78" event={"ID":"fbe49d37-6d34-49a2-8348-8dad98fd6883","Type":"ContainerDied","Data":"cbf51bbd59d171e18d39dde98c1427c4684aee14b319ebfe679024fefd0abdc5"}
Nov 03 19:20:02.698759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.698720    4489 scope.go:122] "RemoveContainer" containerID="98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d"
Nov 03 19:20:02.699012 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:02.698992    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d\": container with ID starting with 98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d not found: ID does not exist" containerID="98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d"
Nov 03 19:20:02.699074 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.699022    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d"} err="failed to get container status \"98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d\": rpc error: code = NotFound desc = could not find container \"98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d\": container with ID starting with 98719d5cdbc4f9aeaab88340eba83ef7258e81913bda4cd9b5e3abc1dae7c20d not found: ID does not exist"
Nov 03 19:20:02.699074 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.699040    4489 scope.go:122] "RemoveContainer" containerID="3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2"
Nov 03 19:20:02.711453 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.711432    4489 scope.go:122] "RemoveContainer" containerID="3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2"
Nov 03 19:20:02.711771 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:02.711749    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2\": container with ID starting with 3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2 not found: ID does not exist" containerID="3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2"
Nov 03 19:20:02.711842 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.711783    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2"} err="failed to get container status \"3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2\": rpc error: code = NotFound desc = could not find container \"3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2\": container with ID starting with 3256f2627f1def701c687a627e93fe04b01d2f46743f5d5e083fe0efd5f41ad2 not found: ID does not exist"
Nov 03 19:20:02.729452 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.729429    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfzjx\" (UniqueName: \"kubernetes.io/projected/fbe49d37-6d34-49a2-8348-8dad98fd6883-kube-api-access-cfzjx\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:02.729452 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.729453    4489 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe49d37-6d34-49a2-8348-8dad98fd6883-plugins-config-volume\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:02.921306 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.921267    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"]
Nov 03 19:20:02.924726 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.924704    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-58ac09-c585d4959-gx5f9"]
Nov 03 19:20:02.994626 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:02.994542    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73eaba1-ec4e-4348-a45c-b364300a95a3" path="/var/lib/kubelet/pods/c73eaba1-ec4e-4348-a45c-b364300a95a3/volumes"
Nov 03 19:20:03.038770 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:03.038734    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"]
Nov 03 19:20:03.042435 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:03.042413    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-58ac09-7846d7dccf-bch78"]
Nov 03 19:20:04.994202 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:04.994172    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe49d37-6d34-49a2-8348-8dad98fd6883" path="/var/lib/kubelet/pods/fbe49d37-6d34-49a2-8348-8dad98fd6883/volumes"
Nov 03 19:20:06.298501 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.298469    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"]
Nov 03 19:20:06.298906 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.298667    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9" podUID="101232cf-aba1-4fef-87ee-4b66ff8f2eeb" containerName="maas-api" containerID="cri-o://99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21" gracePeriod=30
Nov 03 19:20:06.585103 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.585080    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 19:20:06.682800 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.682751    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-openshift-service-ca\") pod \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") "
Nov 03 19:20:06.683775 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.683359    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-metrics-certs\") pod \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") "
Nov 03 19:20:06.683775 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.683483    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-kube-api-access-n4n8n\" (UniqueName: \"kubernetes.io/projected/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-kube-api-access-n4n8n\") pod \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") "
Nov 03 19:20:06.684124 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.684096    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls\") pod \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\" (UID: \"101232cf-aba1-4fef-87ee-4b66ff8f2eeb\") "
Nov 03 19:20:06.684546 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.684507    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-openshift-service-ca" pod "101232cf-aba1-4fef-87ee-4b66ff8f2eeb" (UID: "101232cf-aba1-4fef-87ee-4b66ff8f2eeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:06.684909 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.684702    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:06.687610 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.687572    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls" pod "101232cf-aba1-4fef-87ee-4b66ff8f2eeb" (UID: "101232cf-aba1-4fef-87ee-4b66ff8f2eeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:20:06.687713 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.687668    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-metrics-certs" pod "101232cf-aba1-4fef-87ee-4b66ff8f2eeb" (UID: "101232cf-aba1-4fef-87ee-4b66ff8f2eeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:20:06.687770 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.687717    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-kube-api-access-n4n8n" pod "101232cf-aba1-4fef-87ee-4b66ff8f2eeb" (UID: "101232cf-aba1-4fef-87ee-4b66ff8f2eeb"). InnerVolumeSpecName "kube-api-access-n4n8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:06.712821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.712788    4489 generic.go:356] "Generic (PLEG): container finished" podID="101232cf-aba1-4fef-87ee-4b66ff8f2eeb" containerID="99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21" exitCode=0
Nov 03 19:20:06.712821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.712850    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"
Nov 03 19:20:06.713117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.712967    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9" event={"ID":"101232cf-aba1-4fef-87ee-4b66ff8f2eeb","Type":"ContainerDied","Data":"99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21"}
Nov 03 19:20:06.713117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.713013    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9" event={"ID":"101232cf-aba1-4fef-87ee-4b66ff8f2eeb","Type":"ContainerDied","Data":"b2e5d245c5f556905eb4d91a2d3b90f9de629814723156b67a99bfd71be7fa34"}
Nov 03 19:20:06.713117 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.713043    4489 scope.go:122] "RemoveContainer" containerID="99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21"
Nov 03 19:20:06.727525 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.727490    4489 scope.go:122] "RemoveContainer" containerID="99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21"
Nov 03 19:20:06.728085 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:06.728026    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21\": container with ID starting with 99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21 not found: ID does not exist" containerID="99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21"
Nov 03 19:20:06.728085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.728068    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21"} err="failed to get container status \"99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21\": rpc error: code = NotFound desc = could not find container \"99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21\": container with ID starting with 99afda91fc94bfb745997e65f9781cddf49aab95a787651ec4ef888ab9ad2e21 not found: ID does not exist"
Nov 03 19:20:06.785676 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.785628    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4n8n\" (UniqueName: \"kubernetes.io/projected/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-kube-api-access-n4n8n\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:06.785676 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.785674    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:06.785899 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.785690    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101232cf-aba1-4fef-87ee-4b66ff8f2eeb-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:06.909525 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.909496    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"]
Nov 03 19:20:06.913119 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.913099    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a4cacd-6db957668-5g8z9"]
Nov 03 19:20:06.994041 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:06.994005    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101232cf-aba1-4fef-87ee-4b66ff8f2eeb" path="/var/lib/kubelet/pods/101232cf-aba1-4fef-87ee-4b66ff8f2eeb/volumes"
Nov 03 19:20:15.135065 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:15.135030    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"]
Nov 03 19:20:15.135594 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:15.135258    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy" containerID="cri-o://ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405" gracePeriod=30
Nov 03 19:20:17.529764 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:17.529728    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:20:20.051548 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.051515    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"]
Nov 03 19:20:20.103436 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.103400    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc0f0eec-fnjvg"]
Nov 03 19:20:20.395899 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.395878    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 19:20:20.535825 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.535797    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01570eb7-f2dd-42fc-8b61-35555c4c855e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01570eb7-f2dd-42fc-8b61-35555c4c855e-istiod-ca-cert\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.535981 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.535849    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-kube-api-access-zbxmc\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-kube-api-access-zbxmc\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.535981 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.535896    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-certs\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.535981 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.535961    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-data\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.536092 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536040    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-socket\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.536129 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536103    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-podinfo\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.536225 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536208    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-credential-socket\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.536292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536276    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-token\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-token\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.536358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536280    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-certs" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:20.536358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536305    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-envoy\") pod \"01570eb7-f2dd-42fc-8b61-35555c4c855e\" (UID: \"01570eb7-f2dd-42fc-8b61-35555c4c855e\") "
Nov 03 19:20:20.536483 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536463    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-credential-socket" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:20.536543 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536489    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-socket" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:20.536589 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536540    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01570eb7-f2dd-42fc-8b61-35555c4c855e-istiod-ca-cert" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:20.536639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.536590    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-data" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:20.537037 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.537013    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.537322 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.537042    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.537322 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.537057    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01570eb7-f2dd-42fc-8b61-35555c4c855e-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.537322 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.537072    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.537322 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.537087    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.538468 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.538439    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-kube-api-access-zbxmc" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "kube-api-access-zbxmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:20.538468 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.538463    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-podinfo" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:20:20.538632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.538621    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-envoy" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:20.539047 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.539033    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-token" pod "01570eb7-f2dd-42fc-8b61-35555c4c855e" (UID: "01570eb7-f2dd-42fc-8b61-35555c4c855e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:20.638602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.638525    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.638602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.638549    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.638602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.638560    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbxmc\" (UniqueName: \"kubernetes.io/projected/01570eb7-f2dd-42fc-8b61-35555c4c855e-kube-api-access-zbxmc\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.638602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.638569    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01570eb7-f2dd-42fc-8b61-35555c4c855e-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:20.787030 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.786997    4489 generic.go:356] "Generic (PLEG): container finished" podID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerID="ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405" exitCode=0
Nov 03 19:20:20.787200 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.787049    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"
Nov 03 19:20:20.787200 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.787090    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" event={"ID":"01570eb7-f2dd-42fc-8b61-35555c4c855e","Type":"ContainerDied","Data":"ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405"}
Nov 03 19:20:20.787200 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.787168    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp" event={"ID":"01570eb7-f2dd-42fc-8b61-35555c4c855e","Type":"ContainerDied","Data":"22c8aaf87d016545becdfce965ac29f50d7086acfb2da165336a88ccb0ebb5e0"}
Nov 03 19:20:20.787200 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.787191    4489 scope.go:122] "RemoveContainer" containerID="ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405"
Nov 03 19:20:20.796715 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.796693    4489 scope.go:122] "RemoveContainer" containerID="09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117"
Nov 03 19:20:20.804774 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.804754    4489 scope.go:122] "RemoveContainer" containerID="ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405"
Nov 03 19:20:20.805034 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:20.805014    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405\": container with ID starting with ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405 not found: ID does not exist" containerID="ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405"
Nov 03 19:20:20.805105 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.805048    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405"} err="failed to get container status \"ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405\": rpc error: code = NotFound desc = could not find container \"ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405\": container with ID starting with ab429603cead5facece2a522bd8d74b0df95de2ee05c76dbe6631b4f3077e405 not found: ID does not exist"
Nov 03 19:20:20.805105 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.805074    4489 scope.go:122] "RemoveContainer" containerID="09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117"
Nov 03 19:20:20.805373 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:20.805355    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117\": container with ID starting with 09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117 not found: ID does not exist" containerID="09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117"
Nov 03 19:20:20.805429 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.805382    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117"} err="failed to get container status \"09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117\": rpc error: code = NotFound desc = could not find container \"09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117\": container with ID starting with 09e7dd9191f2784456a23f7b2ef182aa8820fb5739b3e815929e29119a276117 not found: ID does not exist"
Nov 03 19:20:20.958336 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.958304    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"]
Nov 03 19:20:20.995661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:20.995618    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86471317-1765-4670-a922-b60d8f4ae725" path="/var/lib/kubelet/pods/86471317-1765-4670-a922-b60d8f4ae725/volumes"
Nov 03 19:20:21.010439 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:21.010414    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-xmnlp"]
Nov 03 19:20:22.993612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:22.993577    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" path="/var/lib/kubelet/pods/01570eb7-f2dd-42fc-8b61-35555c4c855e/volumes"
Nov 03 19:20:23.295578 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.295494    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"]
Nov 03 19:20:23.295956 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.295937    4489 cpu_manager.go:414] "containerMap: removing container" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.295958    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319b6ca2-9d70-4ac4-b521-c5868508fb51" containerName="istio-proxy"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.295976    4489 cpu_manager.go:414] "containerMap: removing container" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.295983    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.295998    4489 cpu_manager.go:414] "containerMap: removing container" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296006    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296017    4489 cpu_manager.go:414] "containerMap: removing container" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296024    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296035    4489 cpu_manager.go:414] "containerMap: removing container" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:23.296044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296042    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296054    4489 cpu_manager.go:414] "containerMap: removing container" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296061    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296071    4489 cpu_manager.go:414] "containerMap: removing container" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296080    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296094    4489 cpu_manager.go:414] "containerMap: removing container" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296101    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296111    4489 cpu_manager.go:414] "containerMap: removing container" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296118    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296130    4489 cpu_manager.go:414] "containerMap: removing container" podUID="fe7755b4-3a94-46eb-8ff4-3c651c11c230" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296137    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7755b4-3a94-46eb-8ff4-3c651c11c230" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296165    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296172    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296183    4489 cpu_manager.go:414] "containerMap: removing container" podUID="86471317-1765-4670-a922-b60d8f4ae725" containerName="revoke-keys"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296190    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86471317-1765-4670-a922-b60d8f4ae725" containerName="revoke-keys"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296198    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5707bcc4-3660-4c82-9bf5-c2858973132f" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296205    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5707bcc4-3660-4c82-9bf5-c2858973132f" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296219    4489 cpu_manager.go:414] "containerMap: removing container" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296226    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296236    4489 cpu_manager.go:414] "containerMap: removing container" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296242    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296251    4489 cpu_manager.go:414] "containerMap: removing container" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296258    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296266    4489 cpu_manager.go:414] "containerMap: removing container" podUID="65eb0dc6-a873-437e-a958-8a48edbc2158" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296273    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65eb0dc6-a873-437e-a958-8a48edbc2158" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296284    4489 cpu_manager.go:414] "containerMap: removing container" podUID="126f904d-4a17-4e30-95e9-8e343a5f613f" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296291    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126f904d-4a17-4e30-95e9-8e343a5f613f" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296300    4489 cpu_manager.go:414] "containerMap: removing container" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296307    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296318    4489 cpu_manager.go:414] "containerMap: removing container" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296324    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296334    4489 cpu_manager.go:414] "containerMap: removing container" podUID="2d52a40e-0e1d-48ef-91c2-bbf189492e1b" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296341    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d52a40e-0e1d-48ef-91c2-bbf189492e1b" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296353    4489 cpu_manager.go:414] "containerMap: removing container" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296360    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296371    4489 cpu_manager.go:414] "containerMap: removing container" podUID="1bec70e8-d473-4799-ac6d-79fa07e44c99" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296378    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bec70e8-d473-4799-ac6d-79fa07e44c99" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296395    4489 cpu_manager.go:414] "containerMap: removing container" podUID="3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296402    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296413    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296420    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296430    4489 cpu_manager.go:414] "containerMap: removing container" podUID="97ddfc62-ffe9-4e3b-aeee-44aea01eef58" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296439    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ddfc62-ffe9-4e3b-aeee-44aea01eef58" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296450    4489 cpu_manager.go:414] "containerMap: removing container" podUID="c73eaba1-ec4e-4348-a45c-b364300a95a3" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296457    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73eaba1-ec4e-4348-a45c-b364300a95a3" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296471    4489 cpu_manager.go:414] "containerMap: removing container" podUID="101232cf-aba1-4fef-87ee-4b66ff8f2eeb" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296478    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101232cf-aba1-4fef-87ee-4b66ff8f2eeb" containerName="maas-api"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296492    4489 cpu_manager.go:414] "containerMap: removing container" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296499    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296509    4489 cpu_manager.go:414] "containerMap: removing container" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296516    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296526    4489 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296534    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" containerName="payload-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296545    4489 cpu_manager.go:414] "containerMap: removing container" podUID="fbe49d37-6d34-49a2-8348-8dad98fd6883" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296552    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe49d37-6d34-49a2-8348-8dad98fd6883" containerName="payload-pre-processing"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296566    4489 cpu_manager.go:414] "containerMap: removing container" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy"
Nov 03 19:20:23.296547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296572    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296582    4489 cpu_manager.go:414] "containerMap: removing container" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296589    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296601    4489 cpu_manager.go:414] "containerMap: removing container" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296608    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296621    4489 cpu_manager.go:414] "containerMap: removing container" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296628    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296637    4489 cpu_manager.go:414] "containerMap: removing container" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296644    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296657    4489 cpu_manager.go:414] "containerMap: removing container" podUID="44f00105-4d75-443d-bf04-5394b9953c34" containerName="maas-api"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296663    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f00105-4d75-443d-bf04-5394b9953c34" containerName="maas-api"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296770    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7755b4-3a94-46eb-8ff4-3c651c11c230" containerName="maas-api"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296785    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a10f023-91be-4a7b-b2c8-2ff070481e84" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296794    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="86471317-1765-4670-a922-b60d8f4ae725" containerName="revoke-keys"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296803    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5707bcc4-3660-4c82-9bf5-c2858973132f" containerName="maas-api"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296811    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296820    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ea7133-1b1f-44ab-a003-ff63a40a72f9" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296829    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296837    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="65eb0dc6-a873-437e-a958-8a48edbc2158" containerName="payload-pre-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296846    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="126f904d-4a17-4e30-95e9-8e343a5f613f" containerName="maas-api"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296853    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c55759-2371-4366-9749-b11edeb3a7dc" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296863    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296870    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d52a40e-0e1d-48ef-91c2-bbf189492e1b" containerName="payload-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296880    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e29bf55-1ada-4be6-a5ff-af33f146b8a4" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296888    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bec70e8-d473-4799-ac6d-79fa07e44c99" containerName="payload-pre-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296897    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296906    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6a6fb4-9ef1-422b-9d5e-e0a4da966e61" containerName="payload-pre-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296914    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5495cb19-e43c-4c3b-9da5-dcdd3f7b87c7" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296922    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ddfc62-ffe9-4e3b-aeee-44aea01eef58" containerName="payload-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296932    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73eaba1-ec4e-4348-a45c-b364300a95a3" containerName="payload-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296942    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="101232cf-aba1-4fef-87ee-4b66ff8f2eeb" containerName="maas-api"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296950    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296958    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2859a5-32ed-4267-8838-3430edc17c3c" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296966    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="01570eb7-f2dd-42fc-8b61-35555c4c855e" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296975    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2e4eaf-8cea-46bb-9327-9bce2f88e081" containerName="payload-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296985    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe49d37-6d34-49a2-8348-8dad98fd6883" containerName="payload-pre-processing"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.296995    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb54b5f0-0fa8-4f5b-9255-ccc89e323396" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297003    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297013    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c251c64-ee71-4e62-99cc-40ca3e911ab1" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297021    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="635f4eda-d081-47bc-9f53-25559992ea53" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297029    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297036    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f00105-4d75-443d-bf04-5394b9953c34" containerName="maas-api"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297047    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c674b2f-1db3-4a15-9e8f-aa148d40025e" containerName="istio-proxy"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297056    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba917b5c-a85a-4afd-a29c-4d4b6a5ce1a8" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297064    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd329e8-de84-4261-b3fe-8e17b51c550a" containerName="main"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297074    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:23.298905 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.297082    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:23.302106 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.302082    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.354683 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.354649    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"]
Nov 03 19:20:23.403880 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.403850    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9dd9f2\"/\"kube-root-ca.crt\""
Nov 03 19:20:23.404039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.403916    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9dd9f2\"/\"openshift-service-ca.crt\""
Nov 03 19:20:23.404039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.403917    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9dd9f2\"/\"default-dockercfg-9mczp\""
Nov 03 19:20:23.404039 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.403950    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9dd9f2\"/\"test-model-9dd9f2-kserve-self-signed-certs\""
Nov 03 19:20:23.467518 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.467486    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tmp-dir\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.467693 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.467556    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-home\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.467693 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.467644    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrpl\" (UniqueName: \"kubernetes.io/projected/2a01dcb7-4743-434b-b38e-ef4002c13b3e-kube-api-access-skrpl\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.467791 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.467754    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-dshm\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.467791 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.467783    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tls-certs\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.467886 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.467830    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-model-cache\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569277    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tls-certs\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569331    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-model-cache\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569362    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tmp-dir\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569414    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-home\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569567 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569470    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrpl\" (UniqueName: \"kubernetes.io/projected/2a01dcb7-4743-434b-b38e-ef4002c13b3e-kube-api-access-skrpl\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569714 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569597    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-dshm\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569858 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569838    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tmp-dir\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.569919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.569852    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-model-cache\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.570025 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.570007    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-home\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.571856 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.571830    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-dshm\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.571955 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.571939    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tls-certs\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.632712 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.632672    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrpl\" (UniqueName: \"kubernetes.io/projected/2a01dcb7-4743-434b-b38e-ef4002c13b3e-kube-api-access-skrpl\") pod \"test-model-9dd9f2-kserve-85984cf967-4bjxc\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") " pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:23.913801 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:23.913767    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:24.281109 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:20:24.281076    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a01dcb7_4743_434b_b38e_ef4002c13b3e.slice/crio-790880a0a768f48f6f0ffffa8d3cb33441b842a0e9846a2a246fde503dd5bc35 WatchSource:0}: Error finding container 790880a0a768f48f6f0ffffa8d3cb33441b842a0e9846a2a246fde503dd5bc35: Status 404 returned error can't find the container with id 790880a0a768f48f6f0ffffa8d3cb33441b842a0e9846a2a246fde503dd5bc35
Nov 03 19:20:24.284390 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:24.284370    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"]
Nov 03 19:20:24.815425 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:24.815389    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc" event={"ID":"2a01dcb7-4743-434b-b38e-ef4002c13b3e","Type":"ContainerStarted","Data":"080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625"}
Nov 03 19:20:24.815598 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:24.815439    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc" event={"ID":"2a01dcb7-4743-434b-b38e-ef4002c13b3e","Type":"ContainerStarted","Data":"790880a0a768f48f6f0ffffa8d3cb33441b842a0e9846a2a246fde503dd5bc35"}
Nov 03 19:20:24.914294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:24.914262    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:24.937710 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:24.937667    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc" podStartSLOduration=1.937654122 podStartE2EDuration="1.937654122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:24.93596544 +0000 UTC m=+2554.084660011" watchObservedRunningTime="2026-11-03 19:20:24.937654122 +0000 UTC m=+2554.086348693"
Nov 03 19:20:30.544047 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.544013    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"]
Nov 03 19:20:30.544571 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.544286    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6" podUID="b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" containerName="maas-api" containerID="cri-o://424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03" gracePeriod=30
Nov 03 19:20:30.797662 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.797608    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:20:30.850923 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.850886    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-openshift-service-ca\") pod \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") "
Nov 03 19:20:30.851169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.850962    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-kube-api-access-cf2bg\" (UniqueName: \"kubernetes.io/projected/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-kube-api-access-cf2bg\") pod \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") "
Nov 03 19:20:30.851169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.851089    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls\") pod \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") "
Nov 03 19:20:30.857709 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.857663    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-kube-api-access-cf2bg" pod "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" (UID: "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457"). InnerVolumeSpecName "kube-api-access-cf2bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:30.866887 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.866813    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-metrics-certs" pod "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" (UID: "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:20:30.876390 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.876350    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-metrics-certs\") pod \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\" (UID: \"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457\") "
Nov 03 19:20:30.876936 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.876910    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf2bg\" (UniqueName: \"kubernetes.io/projected/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-kube-api-access-cf2bg\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:30.877060 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.876943    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:30.877170 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.877127    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-openshift-service-ca" pod "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" (UID: "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:30.879933 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.879904    4489 generic.go:356] "Generic (PLEG): container finished" podID="b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" containerID="424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03" exitCode=0
Nov 03 19:20:30.880057 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.879991    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6" event={"ID":"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457","Type":"ContainerDied","Data":"424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03"}
Nov 03 19:20:30.880057 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.880044    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6" event={"ID":"b3ec524a-f2fa-4c3c-99c4-78fe30bfa457","Type":"ContainerDied","Data":"d4254749f2defe33ebbd8c999f146873256bd73c465ad96b169414a479424c0f"}
Nov 03 19:20:30.880189 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.880076    4489 scope.go:122] "RemoveContainer" containerID="424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03"
Nov 03 19:20:30.880500 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.880272    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"
Nov 03 19:20:30.887856 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.887819    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls" pod "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" (UID: "b3ec524a-f2fa-4c3c-99c4-78fe30bfa457"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:20:30.901132 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.901111    4489 scope.go:122] "RemoveContainer" containerID="424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03"
Nov 03 19:20:30.901523 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:30.901502    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03\": container with ID starting with 424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03 not found: ID does not exist" containerID="424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03"
Nov 03 19:20:30.901589 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.901537    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03"} err="failed to get container status \"424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03\": rpc error: code = NotFound desc = could not find container \"424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03\": container with ID starting with 424313bbe51c11e2696156c50f1dd7da6268fedfc67e5c1015136e480dc04d03 not found: ID does not exist"
Nov 03 19:20:30.978601 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.978573    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:30.978601 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:30.978601    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:31.305965 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:31.305937    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"]
Nov 03 19:20:31.309420 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:31.309395    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ac09-5dbf886859-vhlx6"]
Nov 03 19:20:32.995356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:32.995322    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" path="/var/lib/kubelet/pods/b3ec524a-f2fa-4c3c-99c4-78fe30bfa457/volumes"
Nov 03 19:20:33.914973 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:33.914939    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:33.927127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:33.927103    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:34.916687 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:34.916658    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:20:43.740946 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.740913    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"]
Nov 03 19:20:43.741369 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741354    4489 cpu_manager.go:414] "containerMap: removing container" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy"
Nov 03 19:20:43.741369 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741366    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy"
Nov 03 19:20:43.741444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741375    4489 cpu_manager.go:414] "containerMap: removing container" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:43.741444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741380    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:43.741444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741393    4489 cpu_manager.go:414] "containerMap: removing container" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:43.741444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741397    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:43.741444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741405    4489 cpu_manager.go:414] "containerMap: removing container" podUID="b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" containerName="maas-api"
Nov 03 19:20:43.741444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741411    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" containerName="maas-api"
Nov 03 19:20:43.741635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741468    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="93065652-85d3-4d61-94dd-af4cdabc7651" containerName="cleanup"
Nov 03 19:20:43.741635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741476    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ec524a-f2fa-4c3c-99c4-78fe30bfa457" containerName="maas-api"
Nov 03 19:20:43.741635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741482    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cbb9d7-bcb3-4b4d-be31-e0640c507f6c" containerName="cleanup"
Nov 03 19:20:43.741635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741491    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e4111b-5349-4747-b39d-adba3dd53310" containerName="istio-proxy"
Nov 03 19:20:43.741635 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.741497    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b14985-244a-4ca7-b977-38cb452421e3" containerName="cleanup"
Nov 03 19:20:43.746009 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.745992    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:43.800218 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.800185    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"]
Nov 03 19:20:43.849843 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.849809    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-942af8\"/\"test-model-942af8-kserve-self-signed-certs\""
Nov 03 19:20:43.850009 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.849880    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-942af8\"/\"default-dockercfg-j8p4r\""
Nov 03 19:20:43.854679 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.854662    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-942af8\"/\"kube-root-ca.crt\""
Nov 03 19:20:43.854757 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.854682    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-942af8\"/\"openshift-service-ca.crt\""
Nov 03 19:20:43.933498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.933465    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-dshm\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:43.933664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.933516    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-model-cache\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:43.933664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.933581    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-home\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:43.933664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.933648    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tmp-dir\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:43.933759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.933686    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tls-certs\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:43.933759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:43.933742    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxxm\" (UniqueName: \"kubernetes.io/projected/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-kube-api-access-dvxxm\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035113 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035026    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-home\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035113 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035084    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tmp-dir\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035371 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035118    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tls-certs\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035371 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035204    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxxm\" (UniqueName: \"kubernetes.io/projected/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-kube-api-access-dvxxm\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035371 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035272    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-dshm\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035371 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035319    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-model-cache\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035575 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035468    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-home\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035575 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035535    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tmp-dir\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.035699 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.035681    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-model-cache\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.037626 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.037608    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-dshm\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.038021 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.037997    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tls-certs\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.098469 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.098432    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxxm\" (UniqueName: \"kubernetes.io/projected/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-kube-api-access-dvxxm\") pod \"test-model-942af8-kserve-6bb8f79d4c-z7zh8\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") " pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.357360 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.357278    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:44.726024 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:20:44.725989    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa0a2408_5a3d_49bc_94f3_eb5d6ea0f74b.slice/crio-ca9db4c97f0ebef1ca07d344caea776145b2c5e1ed74389869017f99d2e11d30 WatchSource:0}: Error finding container ca9db4c97f0ebef1ca07d344caea776145b2c5e1ed74389869017f99d2e11d30: Status 404 returned error can't find the container with id ca9db4c97f0ebef1ca07d344caea776145b2c5e1ed74389869017f99d2e11d30
Nov 03 19:20:44.729041 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.729011    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"]
Nov 03 19:20:44.954705 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.954674    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8" event={"ID":"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b","Type":"ContainerStarted","Data":"67240074f676e6978302775a10add39a0f4dfad391d896bcd478f5ff53394643"}
Nov 03 19:20:44.955102 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:44.954714    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8" event={"ID":"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b","Type":"ContainerStarted","Data":"ca9db4c97f0ebef1ca07d344caea776145b2c5e1ed74389869017f99d2e11d30"}
Nov 03 19:20:45.078493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:45.078398    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8" podStartSLOduration=2.078382955 podStartE2EDuration="2.078382955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:45.076320167 +0000 UTC m=+2574.225014737" watchObservedRunningTime="2026-11-03 19:20:45.078382955 +0000 UTC m=+2574.227077526"
Nov 03 19:20:45.205169 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:45.204865    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"]
Nov 03 19:20:45.209616 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:45.208027    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"]
Nov 03 19:20:45.358236 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:45.358137    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:47.532381 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:20:47.532348    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:20:54.357591 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:54.357558    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:54.370134 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:54.370110    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:20:55.014193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:20:55.014165    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:21:00.229738 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.229692    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" containerID="cri-o://4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42" gracePeriod=45
Nov 03 19:21:00.509179 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.509135    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:21:00.610697 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.610667    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-dshm\") pod \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") "
Nov 03 19:21:00.610864 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.610790    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tmp-dir\") pod \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") "
Nov 03 19:21:00.610864 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.610850    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-kube-api-access-q2zmm\" (UniqueName: \"kubernetes.io/projected/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-kube-api-access-q2zmm\") pod \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") "
Nov 03 19:21:00.610991 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.610880    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-model-cache\") pod \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") "
Nov 03 19:21:00.610991 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.610906    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-home\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-home\") pod \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") "
Nov 03 19:21:00.610991 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.610925    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tls-certs\") pod \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\" (UID: \"7a2a118e-fee9-434d-a571-06d9fb1cbc6b\") "
Nov 03 19:21:00.611127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.611041    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tmp-dir" pod "7a2a118e-fee9-434d-a571-06d9fb1cbc6b" (UID: "7a2a118e-fee9-434d-a571-06d9fb1cbc6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:00.611213 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.611192    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-model-cache" pod "7a2a118e-fee9-434d-a571-06d9fb1cbc6b" (UID: "7a2a118e-fee9-434d-a571-06d9fb1cbc6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:00.611264 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.611210    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-home" pod "7a2a118e-fee9-434d-a571-06d9fb1cbc6b" (UID: "7a2a118e-fee9-434d-a571-06d9fb1cbc6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:00.611358 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.611337    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:00.611397 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.611357    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:00.611397 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.611376    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:00.612953 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.612930    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-dshm" pod "7a2a118e-fee9-434d-a571-06d9fb1cbc6b" (UID: "7a2a118e-fee9-434d-a571-06d9fb1cbc6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:00.613102 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.613083    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-kube-api-access-q2zmm" pod "7a2a118e-fee9-434d-a571-06d9fb1cbc6b" (UID: "7a2a118e-fee9-434d-a571-06d9fb1cbc6b"). InnerVolumeSpecName "kube-api-access-q2zmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:00.613231 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.613213    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tls-certs" pod "7a2a118e-fee9-434d-a571-06d9fb1cbc6b" (UID: "7a2a118e-fee9-434d-a571-06d9fb1cbc6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:21:00.712740 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.712701    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zmm\" (UniqueName: \"kubernetes.io/projected/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-kube-api-access-q2zmm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:00.712740 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.712743    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:00.712924 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.712758    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a2a118e-fee9-434d-a571-06d9fb1cbc6b-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:00.978452 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:00.978399    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main" containerID="cri-o://7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d" gracePeriod=45
Nov 03 19:21:01.032826 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.032797    4489 generic.go:356] "Generic (PLEG): container finished" podID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerID="4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42" exitCode=0
Nov 03 19:21:01.032932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.032838    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"
Nov 03 19:21:01.032932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.032859    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" event={"ID":"7a2a118e-fee9-434d-a571-06d9fb1cbc6b","Type":"ContainerDied","Data":"4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42"}
Nov 03 19:21:01.032932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.032893    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" event={"ID":"7a2a118e-fee9-434d-a571-06d9fb1cbc6b","Type":"ContainerDied","Data":"f06ef22c6c73225585eeefbf2270d36845266dd439b833694efebdb472f4eb4d"}
Nov 03 19:21:01.032932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.032915    4489 scope.go:122] "RemoveContainer" containerID="4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42"
Nov 03 19:21:01.091409 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.091314    4489 scope.go:122] "RemoveContainer" containerID="4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42"
Nov 03 19:21:01.091738 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:21:01.091712    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42\": container with ID starting with 4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42 not found: ID does not exist" containerID="4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42"
Nov 03 19:21:01.091791 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.091750    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42"} err="failed to get container status \"4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42\": rpc error: code = NotFound desc = could not find container \"4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42\": container with ID starting with 4764b92bdd2907add0af4b918172a4528dfd74a1b5154a132302dbaae7ca6a42 not found: ID does not exist"
Nov 03 19:21:01.206581 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.206549    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"]
Nov 03 19:21:01.210359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.210335    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p"]
Nov 03 19:21:01.219285 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.219262    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:21:01.317910 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.317880    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-dshm\") pod \"f126eb0b-1e06-4502-b8c5-c802a034d160\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") "
Nov 03 19:21:01.318395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.317986    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-home\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-home\") pod \"f126eb0b-1e06-4502-b8c5-c802a034d160\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") "
Nov 03 19:21:01.318395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318013    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f126eb0b-1e06-4502-b8c5-c802a034d160-kube-api-access-xrn6g\" (UniqueName: \"kubernetes.io/projected/f126eb0b-1e06-4502-b8c5-c802a034d160-kube-api-access-xrn6g\") pod \"f126eb0b-1e06-4502-b8c5-c802a034d160\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") "
Nov 03 19:21:01.318395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318037    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-tmp-dir\") pod \"f126eb0b-1e06-4502-b8c5-c802a034d160\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") "
Nov 03 19:21:01.318395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318061    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-model-cache\") pod \"f126eb0b-1e06-4502-b8c5-c802a034d160\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") "
Nov 03 19:21:01.318395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318129    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f126eb0b-1e06-4502-b8c5-c802a034d160-tls-certs\" (UniqueName: \"kubernetes.io/secret/f126eb0b-1e06-4502-b8c5-c802a034d160-tls-certs\") pod \"f126eb0b-1e06-4502-b8c5-c802a034d160\" (UID: \"f126eb0b-1e06-4502-b8c5-c802a034d160\") "
Nov 03 19:21:01.318395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318315    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-tmp-dir" pod "f126eb0b-1e06-4502-b8c5-c802a034d160" (UID: "f126eb0b-1e06-4502-b8c5-c802a034d160"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:01.318395 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318327    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-home" pod "f126eb0b-1e06-4502-b8c5-c802a034d160" (UID: "f126eb0b-1e06-4502-b8c5-c802a034d160"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:01.318932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318409    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-model-cache" pod "f126eb0b-1e06-4502-b8c5-c802a034d160" (UID: "f126eb0b-1e06-4502-b8c5-c802a034d160"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:01.318932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318519    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:01.318932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318537    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:01.318932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.318550    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:01.320217 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.320193    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-dshm" pod "f126eb0b-1e06-4502-b8c5-c802a034d160" (UID: "f126eb0b-1e06-4502-b8c5-c802a034d160"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:01.320315 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.320290    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f126eb0b-1e06-4502-b8c5-c802a034d160-kube-api-access-xrn6g" pod "f126eb0b-1e06-4502-b8c5-c802a034d160" (UID: "f126eb0b-1e06-4502-b8c5-c802a034d160"). InnerVolumeSpecName "kube-api-access-xrn6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:01.320379 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.320365    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f126eb0b-1e06-4502-b8c5-c802a034d160-tls-certs" pod "f126eb0b-1e06-4502-b8c5-c802a034d160" (UID: "f126eb0b-1e06-4502-b8c5-c802a034d160"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:21:01.419405 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.419374    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrn6g\" (UniqueName: \"kubernetes.io/projected/f126eb0b-1e06-4502-b8c5-c802a034d160-kube-api-access-xrn6g\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:01.419405 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.419400    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126eb0b-1e06-4502-b8c5-c802a034d160-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:01.419405 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.419411    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126eb0b-1e06-4502-b8c5-c802a034d160-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:01.502230 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:01.502132    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/distinct-w4-2e052a-kserve-5fbf8df994-mg46p" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:21:02.039347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.039315    4489 generic.go:356] "Generic (PLEG): container finished" podID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerID="7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d" exitCode=0
Nov 03 19:21:02.039590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.039395    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"
Nov 03 19:21:02.039590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.039401    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" event={"ID":"f126eb0b-1e06-4502-b8c5-c802a034d160","Type":"ContainerDied","Data":"7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d"}
Nov 03 19:21:02.039590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.039440    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c" event={"ID":"f126eb0b-1e06-4502-b8c5-c802a034d160","Type":"ContainerDied","Data":"4723feb34e316cbb5cfec407f77a0d62ae86db92ea492c596c901fb8952a7a53"}
Nov 03 19:21:02.039590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.039457    4489 scope.go:122] "RemoveContainer" containerID="7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d"
Nov 03 19:21:02.049325 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.049307    4489 scope.go:122] "RemoveContainer" containerID="7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d"
Nov 03 19:21:02.049614 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:21:02.049588    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d\": container with ID starting with 7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d not found: ID does not exist" containerID="7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d"
Nov 03 19:21:02.049715 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.049625    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d"} err="failed to get container status \"7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d\": rpc error: code = NotFound desc = could not find container \"7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d\": container with ID starting with 7f6ceb018aeed558b851f4b4e04b42fd3c4f6d13c05a1c07ed1828b8f385fe1d not found: ID does not exist"
Nov 03 19:21:02.212020 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.211972    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"]
Nov 03 19:21:02.216061 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.216031    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/distinct-2-w4-2e052a-kserve-8df5874cc-5gm8c"]
Nov 03 19:21:02.994694 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.994662    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" path="/var/lib/kubelet/pods/7a2a118e-fee9-434d-a571-06d9fb1cbc6b/volumes"
Nov 03 19:21:02.995165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:02.995125    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" path="/var/lib/kubelet/pods/f126eb0b-1e06-4502-b8c5-c802a034d160/volumes"
Nov 03 19:21:12.659738 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.659707    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-54db2c"]
Nov 03 19:21:12.660127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.660118    4489 cpu_manager.go:414] "containerMap: removing container" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main"
Nov 03 19:21:12.660193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.660130    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main"
Nov 03 19:21:12.660193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.660181    4489 cpu_manager.go:414] "containerMap: removing container" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main"
Nov 03 19:21:12.660193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.660189    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main"
Nov 03 19:21:12.660294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.660272    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="f126eb0b-1e06-4502-b8c5-c802a034d160" containerName="main"
Nov 03 19:21:12.660294 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.660286    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2a118e-fee9-434d-a571-06d9fb1cbc6b" containerName="main"
Nov 03 19:21:12.664809 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.664790    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:12.713738 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.713712    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-54db2c"]
Nov 03 19:21:12.745560 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.745531    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qgk\" (UniqueName: \"kubernetes.io/projected/09fc0272-aecb-46f6-8868-0d307692ad5c-kube-api-access-49qgk\") pod \"test-curl-7315-54db2c\" (UID: \"09fc0272-aecb-46f6-8868-0d307692ad5c\") " pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:12.763436 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.763416    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zmjtl\""
Nov 03 19:21:12.846442 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.846415    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qgk\" (UniqueName: \"kubernetes.io/projected/09fc0272-aecb-46f6-8868-0d307692ad5c-kube-api-access-49qgk\") pod \"test-curl-7315-54db2c\" (UID: \"09fc0272-aecb-46f6-8868-0d307692ad5c\") " pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:12.909623 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.909594    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qgk\" (UniqueName: \"kubernetes.io/projected/09fc0272-aecb-46f6-8868-0d307692ad5c-kube-api-access-49qgk\") pod \"test-curl-7315-54db2c\" (UID: \"09fc0272-aecb-46f6-8868-0d307692ad5c\") " pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:12.976005 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:12.975950    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:13.344702 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:21:13.344672    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09fc0272_aecb_46f6_8868_0d307692ad5c.slice/crio-6c9ca62fd687d0bbe81393f2f0362ce1acf1c4a49bcec5f425671caeb1c5fac6 WatchSource:0}: Error finding container 6c9ca62fd687d0bbe81393f2f0362ce1acf1c4a49bcec5f425671caeb1c5fac6: Status 404 returned error can't find the container with id 6c9ca62fd687d0bbe81393f2f0362ce1acf1c4a49bcec5f425671caeb1c5fac6
Nov 03 19:21:13.348067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:13.348047    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-54db2c"]
Nov 03 19:21:14.105374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:14.105325    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-54db2c" event={"ID":"09fc0272-aecb-46f6-8868-0d307692ad5c","Type":"ContainerStarted","Data":"78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae"}
Nov 03 19:21:14.105759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:14.105382    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-54db2c" event={"ID":"09fc0272-aecb-46f6-8868-0d307692ad5c","Type":"ContainerStarted","Data":"6c9ca62fd687d0bbe81393f2f0362ce1acf1c4a49bcec5f425671caeb1c5fac6"}
Nov 03 19:21:14.224531 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:14.224477    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-54db2c" podStartSLOduration=2.224462269 podStartE2EDuration="2.224462269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:14.222519128 +0000 UTC m=+2603.371213717" watchObservedRunningTime="2026-11-03 19:21:14.224462269 +0000 UTC m=+2603.373156839"
Nov 03 19:21:15.769262 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:15.769226    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-54db2c"]
Nov 03 19:21:15.772103 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:15.772074    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-54db2c"]
Nov 03 19:21:16.116507 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:16.116421    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-54db2c" podUID="09fc0272-aecb-46f6-8868-0d307692ad5c" containerName="test-curl-7315-54db2c" containerID="cri-o://78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae" gracePeriod=2
Nov 03 19:21:16.169890 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:21:16.169849    4489 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-54db2c\" is forbidden: User \"system:node:ip-10-0-143-45.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-45.us-west-2.compute.internal' and this object" podUID="09fc0272-aecb-46f6-8868-0d307692ad5c" pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:17.531709 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:21:17.531673    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:21:18.351499 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:18.351470    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:18.511008 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:18.510977    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fc0272-aecb-46f6-8868-0d307692ad5c-kube-api-access-49qgk\" (UniqueName: \"kubernetes.io/projected/09fc0272-aecb-46f6-8868-0d307692ad5c-kube-api-access-49qgk\") pod \"09fc0272-aecb-46f6-8868-0d307692ad5c\" (UID: \"09fc0272-aecb-46f6-8868-0d307692ad5c\") "
Nov 03 19:21:18.513279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:18.513241    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fc0272-aecb-46f6-8868-0d307692ad5c-kube-api-access-49qgk" pod "09fc0272-aecb-46f6-8868-0d307692ad5c" (UID: "09fc0272-aecb-46f6-8868-0d307692ad5c"). InnerVolumeSpecName "kube-api-access-49qgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:18.611945 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:18.611911    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qgk\" (UniqueName: \"kubernetes.io/projected/09fc0272-aecb-46f6-8868-0d307692ad5c-kube-api-access-49qgk\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:18.994239 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:18.994194    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fc0272-aecb-46f6-8868-0d307692ad5c" path="/var/lib/kubelet/pods/09fc0272-aecb-46f6-8868-0d307692ad5c/volumes"
Nov 03 19:21:19.132337 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:19.132304    4489 generic.go:356] "Generic (PLEG): container finished" podID="09fc0272-aecb-46f6-8868-0d307692ad5c" containerID="78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae" exitCode=137
Nov 03 19:21:19.132505 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:19.132354    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-54db2c"
Nov 03 19:21:19.132505 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:19.132405    4489 scope.go:122] "RemoveContainer" containerID="78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae"
Nov 03 19:21:19.140543 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:19.140524    4489 scope.go:122] "RemoveContainer" containerID="78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae"
Nov 03 19:21:19.140858 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:21:19.140837    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae\": container with ID starting with 78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae not found: ID does not exist" containerID="78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae"
Nov 03 19:21:19.140919 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:19.140868    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae"} err="failed to get container status \"78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae\": rpc error: code = NotFound desc = could not find container \"78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae\": container with ID starting with 78766822df9321366ac620ce0a1cf86983b415a4e3644ddb27ac050921d4d7ae not found: ID does not exist"
Nov 03 19:21:27.854872 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.854837    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"]
Nov 03 19:21:27.855528 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.855505    4489 cpu_manager.go:414] "containerMap: removing container" podUID="09fc0272-aecb-46f6-8868-0d307692ad5c" containerName="test-curl-7315-54db2c"
Nov 03 19:21:27.855572 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.855532    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fc0272-aecb-46f6-8868-0d307692ad5c" containerName="test-curl-7315-54db2c"
Nov 03 19:21:27.855657 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.855644    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fc0272-aecb-46f6-8868-0d307692ad5c" containerName="test-curl-7315-54db2c"
Nov 03 19:21:27.865643 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.865616    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:27.913540 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.913504    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"]
Nov 03 19:21:27.916321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.916290    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:27.916456 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.916407    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:27.916539 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.916526    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/ed198511-d5dd-4f0c-afd8-672df97752f7-kube-api-access-m2gnq\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:27.961620 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.961595    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:21:27.961853 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:27.961836    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\""
Nov 03 19:21:28.017646 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.017616    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:28.017773 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.017737    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/ed198511-d5dd-4f0c-afd8-672df97752f7-kube-api-access-m2gnq\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:28.017829 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.017817    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:28.018461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.018443    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:28.020188 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.020131    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:28.081065 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.081043    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/ed198511-d5dd-4f0c-afd8-672df97752f7-kube-api-access-m2gnq\") pod \"maas-api-revoke-sub-57d24373-s72d4\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:28.179322 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.179293    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:28.545615 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:21:28.545585    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded198511_d5dd_4f0c_afd8_672df97752f7.slice/crio-c25a6db851d40cd272f62fd869a62c4a4a934787979ae5ca0431e72f67bc4b26 WatchSource:0}: Error finding container c25a6db851d40cd272f62fd869a62c4a4a934787979ae5ca0431e72f67bc4b26: Status 404 returned error can't find the container with id c25a6db851d40cd272f62fd869a62c4a4a934787979ae5ca0431e72f67bc4b26
Nov 03 19:21:28.548720 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:28.548699    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"]
Nov 03 19:21:29.184324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:29.184286    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4" event={"ID":"ed198511-d5dd-4f0c-afd8-672df97752f7","Type":"ContainerStarted","Data":"a56bbb0ce9ab07e825860f5e44ca9c2f1fbfe2ebe90c09e28184e490a2a050b6"}
Nov 03 19:21:29.184756 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:29.184342    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4" event={"ID":"ed198511-d5dd-4f0c-afd8-672df97752f7","Type":"ContainerStarted","Data":"c25a6db851d40cd272f62fd869a62c4a4a934787979ae5ca0431e72f67bc4b26"}
Nov 03 19:21:29.305015 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:29.304953    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4" podStartSLOduration=2.30493934 podStartE2EDuration="2.30493934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:21:29.302948903 +0000 UTC m=+2618.451643474" watchObservedRunningTime="2026-11-03 19:21:29.30493934 +0000 UTC m=+2618.453633911"
Nov 03 19:21:30.190393 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:30.190357    4489 generic.go:356] "Generic (PLEG): container finished" podID="ed198511-d5dd-4f0c-afd8-672df97752f7" containerID="a56bbb0ce9ab07e825860f5e44ca9c2f1fbfe2ebe90c09e28184e490a2a050b6" exitCode=0
Nov 03 19:21:30.190818 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:30.190408    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4" event={"ID":"ed198511-d5dd-4f0c-afd8-672df97752f7","Type":"ContainerDied","Data":"a56bbb0ce9ab07e825860f5e44ca9c2f1fbfe2ebe90c09e28184e490a2a050b6"}
Nov 03 19:21:31.334972 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.334941    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:31.467658 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.467581    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-xks-ca\") pod \"ed198511-d5dd-4f0c-afd8-672df97752f7\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") "
Nov 03 19:21:31.467658 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.467621    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed198511-d5dd-4f0c-afd8-672df97752f7-kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/ed198511-d5dd-4f0c-afd8-672df97752f7-kube-api-access-m2gnq\") pod \"ed198511-d5dd-4f0c-afd8-672df97752f7\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") "
Nov 03 19:21:31.467889 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.467690    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-openshift-ca\") pod \"ed198511-d5dd-4f0c-afd8-672df97752f7\" (UID: \"ed198511-d5dd-4f0c-afd8-672df97752f7\") "
Nov 03 19:21:31.468087 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.468065    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-openshift-ca" pod "ed198511-d5dd-4f0c-afd8-672df97752f7" (UID: "ed198511-d5dd-4f0c-afd8-672df97752f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:31.469850 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.469815    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-xks-ca" pod "ed198511-d5dd-4f0c-afd8-672df97752f7" (UID: "ed198511-d5dd-4f0c-afd8-672df97752f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:21:31.469976 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.469914    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed198511-d5dd-4f0c-afd8-672df97752f7-kube-api-access-m2gnq" pod "ed198511-d5dd-4f0c-afd8-672df97752f7" (UID: "ed198511-d5dd-4f0c-afd8-672df97752f7"). InnerVolumeSpecName "kube-api-access-m2gnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:31.568632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.568601    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-xks-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:31.568632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.568629    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/ed198511-d5dd-4f0c-afd8-672df97752f7-kube-api-access-m2gnq\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:31.568632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:31.568639    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed198511-d5dd-4f0c-afd8-672df97752f7-maas-api-openshift-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.202000 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:32.201967    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"
Nov 03 19:21:32.202000 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:32.201989    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4" event={"ID":"ed198511-d5dd-4f0c-afd8-672df97752f7","Type":"ContainerDied","Data":"c25a6db851d40cd272f62fd869a62c4a4a934787979ae5ca0431e72f67bc4b26"}
Nov 03 19:21:32.202275 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:32.202039    4489 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c25a6db851d40cd272f62fd869a62c4a4a934787979ae5ca0431e72f67bc4b26"
Nov 03 19:21:47.529835 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:21:47.529782    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:21:51.717669 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:51.717636    4489 scope.go:122] "RemoveContainer" containerID="75b4ea2030af8f0db1b65b8889e49660864d9279bef04f25994139485f56f6ba"
Nov 03 19:21:57.210931 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:57.210898    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"]
Nov 03 19:21:57.211441 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:21:57.211086    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy" containerID="cri-o://1ba32c925cbd191f35208cb5d4de3907bc9b8b87ce28d2539fdf769216b7b248" gracePeriod=30
Nov 03 19:22:02.364426 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.364392    4489 generic.go:356] "Generic (PLEG): container finished" podID="787654a0-d74e-40dd-9641-db7df5d589cd" containerID="1ba32c925cbd191f35208cb5d4de3907bc9b8b87ce28d2539fdf769216b7b248" exitCode=0
Nov 03 19:22:02.364833 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.364470    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" event={"ID":"787654a0-d74e-40dd-9641-db7df5d589cd","Type":"ContainerDied","Data":"1ba32c925cbd191f35208cb5d4de3907bc9b8b87ce28d2539fdf769216b7b248"}
Nov 03 19:22:02.464184 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.464132    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 19:22:02.587243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587132    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-kube-api-access-68cnd\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-kube-api-access-68cnd\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587196    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-envoy\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587248    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-credential-socket\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587282    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-data\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587319    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-certs\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587341    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/787654a0-d74e-40dd-9641-db7df5d589cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787654a0-d74e-40dd-9641-db7df5d589cd-istio-podinfo\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587370    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-istio-token\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-istio-token\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587423    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/787654a0-d74e-40dd-9641-db7df5d589cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787654a0-d74e-40dd-9641-db7df5d589cd-istiod-ca-cert\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587471    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-socket\") pod \"787654a0-d74e-40dd-9641-db7df5d589cd\" (UID: \"787654a0-d74e-40dd-9641-db7df5d589cd\") "
Nov 03 19:22:02.587853 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587595    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-data" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:02.587853 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587615    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-credential-socket" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:02.587853 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587608    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-certs" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:02.587853 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587826    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-socket" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:02.588270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587901    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.588270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587944    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787654a0-d74e-40dd-9641-db7df5d589cd-istiod-ca-cert" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:22:02.588270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.587949    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.588270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.588103    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.589644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.589616    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-envoy" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:02.589644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.589625    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/787654a0-d74e-40dd-9641-db7df5d589cd-istio-podinfo" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:22:02.589816 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.589722    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-kube-api-access-68cnd" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "kube-api-access-68cnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:02.589867 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.589815    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-istio-token" pod "787654a0-d74e-40dd-9641-db7df5d589cd" (UID: "787654a0-d74e-40dd-9641-db7df5d589cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:02.689016 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.688983    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68cnd\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-kube-api-access-68cnd\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.689016 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.689015    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.689250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.689027    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787654a0-d74e-40dd-9641-db7df5d589cd-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.689250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.689040    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787654a0-d74e-40dd-9641-db7df5d589cd-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.689250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.689050    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787654a0-d74e-40dd-9641-db7df5d589cd-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:02.689250 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:02.689062    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787654a0-d74e-40dd-9641-db7df5d589cd-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:03.371256 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:03.371182    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"
Nov 03 19:22:03.371677 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:03.371175    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw" event={"ID":"787654a0-d74e-40dd-9641-db7df5d589cd","Type":"ContainerDied","Data":"6fe33a0c6dcd22618994ee24ab82b8daaae830f31f6efe1e3b285cf802db1ec4"}
Nov 03 19:22:03.371677 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:03.371312    4489 scope.go:122] "RemoveContainer" containerID="1ba32c925cbd191f35208cb5d4de3907bc9b8b87ce28d2539fdf769216b7b248"
Nov 03 19:22:03.543157 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:03.543120    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"]
Nov 03 19:22:03.594695 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:03.594658    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-8m7jw"]
Nov 03 19:22:05.011297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:05.011258    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" path="/var/lib/kubelet/pods/787654a0-d74e-40dd-9641-db7df5d589cd/volumes"
Nov 03 19:22:12.285773 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:12.285737    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"]
Nov 03 19:22:12.286271 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:12.286063    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy" containerID="cri-o://3a91f2f197df392543f6bad4de086fc85e104f4bd187cb02d061b8ae02ea7cf6" gracePeriod=30
Nov 03 19:22:17.442598 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.442566    4489 generic.go:356] "Generic (PLEG): container finished" podID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerID="3a91f2f197df392543f6bad4de086fc85e104f4bd187cb02d061b8ae02ea7cf6" exitCode=0
Nov 03 19:22:17.443056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.442643    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" event={"ID":"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22","Type":"ContainerDied","Data":"3a91f2f197df392543f6bad4de086fc85e104f4bd187cb02d061b8ae02ea7cf6"}
Nov 03 19:22:17.443056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.442724    4489 scope.go:122] "RemoveContainer" containerID="92b017934b83edd560a802dbf756cdf0c0605650059e0594ccfd0ebf5b48e349"
Nov 03 19:22:17.530722 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:22:17.530690    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:22:17.535856 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.535833    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 19:22:17.650629 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650562    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istiod-ca-cert\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.650629 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650598    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-socket\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.650629 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650617    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-podinfo\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.650843 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650803    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-data\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.650900 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650841    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-kube-api-access-dw84c\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-kube-api-access-dw84c\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.650900 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650872    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-credential-socket\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.651013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650894    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-socket" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:17.651013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.650948    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istiod-ca-cert" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:22:17.651112 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651099    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-certs\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.651201 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651104    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-data" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:17.651201 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651124    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-credential-socket" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:17.651201 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651138    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-envoy\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.651357 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651212    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-token\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-token\") pod \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\" (UID: \"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22\") "
Nov 03 19:22:17.651357 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651321    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-certs" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:17.651668 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651629    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.651948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651680    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.651948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651806    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.651948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651845    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.651948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.651859    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.652998 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.652977    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-podinfo" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:22:17.653122 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.653004    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-kube-api-access-dw84c" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "kube-api-access-dw84c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:17.653324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.653292    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-envoy" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:17.653440 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.653359    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-token" pod "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" (UID: "1c7e4ba8-cc0a-427b-aea8-cb387ad31a22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:17.752849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.752817    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.752849 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.752845    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.753034 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.752857    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:17.753034 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:17.752869    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw84c\" (UniqueName: \"kubernetes.io/projected/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22-kube-api-access-dw84c\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:18.449838 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:18.449808    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"
Nov 03 19:22:18.450359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:18.449839    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf" event={"ID":"1c7e4ba8-cc0a-427b-aea8-cb387ad31a22","Type":"ContainerDied","Data":"7bcdfa4c82734c85daf26bdaa9572659a924dfc86c76ccf957dd7f48ab285a40"}
Nov 03 19:22:18.450359 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:18.449938    4489 scope.go:122] "RemoveContainer" containerID="3a91f2f197df392543f6bad4de086fc85e104f4bd187cb02d061b8ae02ea7cf6"
Nov 03 19:22:18.676290 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:18.676255    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"]
Nov 03 19:22:18.681524 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:18.681498    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-7fnvf"]
Nov 03 19:22:18.994283 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:22:18.994248    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" path="/var/lib/kubelet/pods/1c7e4ba8-cc0a-427b-aea8-cb387ad31a22/volumes"
Nov 03 19:22:47.533051 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:22:47.533012    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:23:17.530542 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:23:17.530505    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:23:39.355990 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:39.355904    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"]
Nov 03 19:23:47.532072 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:23:47.532036    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:23:54.376786 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.376723    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc" podUID="2a01dcb7-4743-434b-b38e-ef4002c13b3e" containerName="main" containerID="cri-o://080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625" gracePeriod=45
Nov 03 19:23:54.688769 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.688746    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:23:54.796576 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796543    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tmp-dir\") pod \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") "
Nov 03 19:23:54.796793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796660    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-model-cache\") pod \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") "
Nov 03 19:23:54.796793 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796740    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-home\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-home\") pod \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") "
Nov 03 19:23:54.796913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796789    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-dshm\") pod \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") "
Nov 03 19:23:54.796913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796828    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tmp-dir" pod "2a01dcb7-4743-434b-b38e-ef4002c13b3e" (UID: "2a01dcb7-4743-434b-b38e-ef4002c13b3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:54.796913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796842    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tls-certs\") pod \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") "
Nov 03 19:23:54.796913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796873    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a01dcb7-4743-434b-b38e-ef4002c13b3e-kube-api-access-skrpl\" (UniqueName: \"kubernetes.io/projected/2a01dcb7-4743-434b-b38e-ef4002c13b3e-kube-api-access-skrpl\") pod \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\" (UID: \"2a01dcb7-4743-434b-b38e-ef4002c13b3e\") "
Nov 03 19:23:54.797103 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.796931    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-model-cache" pod "2a01dcb7-4743-434b-b38e-ef4002c13b3e" (UID: "2a01dcb7-4743-434b-b38e-ef4002c13b3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:54.797103 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.797017    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-home" pod "2a01dcb7-4743-434b-b38e-ef4002c13b3e" (UID: "2a01dcb7-4743-434b-b38e-ef4002c13b3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:54.797322 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.797304    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:54.797391 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.797327    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:54.797391 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.797341    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:54.799198 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.799176    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tls-certs" pod "2a01dcb7-4743-434b-b38e-ef4002c13b3e" (UID: "2a01dcb7-4743-434b-b38e-ef4002c13b3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:23:54.799279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.799201    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a01dcb7-4743-434b-b38e-ef4002c13b3e-kube-api-access-skrpl" pod "2a01dcb7-4743-434b-b38e-ef4002c13b3e" (UID: "2a01dcb7-4743-434b-b38e-ef4002c13b3e"). InnerVolumeSpecName "kube-api-access-skrpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:54.799279 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.799200    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-dshm" pod "2a01dcb7-4743-434b-b38e-ef4002c13b3e" (UID: "2a01dcb7-4743-434b-b38e-ef4002c13b3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:54.898542 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.898503    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a01dcb7-4743-434b-b38e-ef4002c13b3e-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:54.898542 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.898537    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a01dcb7-4743-434b-b38e-ef4002c13b3e-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:54.898542 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.898547    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skrpl\" (UniqueName: \"kubernetes.io/projected/2a01dcb7-4743-434b-b38e-ef4002c13b3e-kube-api-access-skrpl\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:54.923194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.923082    4489 generic.go:356] "Generic (PLEG): container finished" podID="2a01dcb7-4743-434b-b38e-ef4002c13b3e" containerID="080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625" exitCode=0
Nov 03 19:23:54.923194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.923161    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"
Nov 03 19:23:54.923194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.923177    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc" event={"ID":"2a01dcb7-4743-434b-b38e-ef4002c13b3e","Type":"ContainerDied","Data":"080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625"}
Nov 03 19:23:54.923194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.923228    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc" event={"ID":"2a01dcb7-4743-434b-b38e-ef4002c13b3e","Type":"ContainerDied","Data":"790880a0a768f48f6f0ffffa8d3cb33441b842a0e9846a2a246fde503dd5bc35"}
Nov 03 19:23:54.923521 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.923248    4489 scope.go:122] "RemoveContainer" containerID="080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625"
Nov 03 19:23:54.932672 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.932652    4489 scope.go:122] "RemoveContainer" containerID="080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625"
Nov 03 19:23:54.932982 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:23:54.932960    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625\": container with ID starting with 080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625 not found: ID does not exist" containerID="080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625"
Nov 03 19:23:54.933073 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:54.932992    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625"} err="failed to get container status \"080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625\": rpc error: code = NotFound desc = could not find container \"080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625\": container with ID starting with 080c6752ee60a6fe01b9129b80b837969dbb4a7f72a69248a6b72bb6fe493625 not found: ID does not exist"
Nov 03 19:23:55.093298 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:55.093259    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"]
Nov 03 19:23:55.097400 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:55.097378    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9dd9f2/test-model-9dd9f2-kserve-85984cf967-4bjxc"]
Nov 03 19:23:56.994345 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:23:56.994312    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a01dcb7-4743-434b-b38e-ef4002c13b3e" path="/var/lib/kubelet/pods/2a01dcb7-4743-434b-b38e-ef4002c13b3e/volumes"
Nov 03 19:24:05.866357 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:05.866326    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"]
Nov 03 19:24:17.531772 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:24:17.531732    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:24:20.886300 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:20.886258    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8" podUID="fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" containerName="main" containerID="cri-o://67240074f676e6978302775a10add39a0f4dfad391d896bcd478f5ff53394643" gracePeriod=45
Nov 03 19:24:21.004035 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.003971    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8" podUID="fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": dial tcp 10.133.0.103:8000: connect: connection refused"
Nov 03 19:24:21.073672 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.073636    4489 generic.go:356] "Generic (PLEG): container finished" podID="fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" containerID="67240074f676e6978302775a10add39a0f4dfad391d896bcd478f5ff53394643" exitCode=0
Nov 03 19:24:21.073834 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.073709    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8" event={"ID":"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b","Type":"ContainerDied","Data":"67240074f676e6978302775a10add39a0f4dfad391d896bcd478f5ff53394643"}
Nov 03 19:24:21.168278 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.168253    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:24:21.272687 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.272661    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-kube-api-access-dvxxm\" (UniqueName: \"kubernetes.io/projected/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-kube-api-access-dvxxm\") pod \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") "
Nov 03 19:24:21.272848 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.272712    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tmp-dir\") pod \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") "
Nov 03 19:24:21.272921 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.272878    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-home\") pod \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") "
Nov 03 19:24:21.272986 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.272944    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tmp-dir" pod "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" (UID: "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:21.272986 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.272979    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-dshm\") pod \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") "
Nov 03 19:24:21.273094 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.273008    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tls-certs\") pod \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") "
Nov 03 19:24:21.273094 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.273037    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-model-cache\") pod \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\" (UID: \"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b\") "
Nov 03 19:24:21.273094 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.273086    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-home" pod "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" (UID: "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:21.273387 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.273366    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-model-cache" pod "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" (UID: "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:21.273461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.273444    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:21.273461 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.273457    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:21.273528 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.273466    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:21.274953 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.274928    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-kube-api-access-dvxxm" pod "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" (UID: "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b"). InnerVolumeSpecName "kube-api-access-dvxxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:21.275249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.275228    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tls-certs" pod "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" (UID: "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:24:21.275398 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.275380    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-dshm" pod "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" (UID: "fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:21.374165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.374121    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxxm\" (UniqueName: \"kubernetes.io/projected/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-kube-api-access-dvxxm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:21.374165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.374168    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:21.374363 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:21.374178    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:22.081598 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:22.081567    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8" event={"ID":"fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b","Type":"ContainerDied","Data":"ca9db4c97f0ebef1ca07d344caea776145b2c5e1ed74389869017f99d2e11d30"}
Nov 03 19:24:22.081988 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:22.081612    4489 scope.go:122] "RemoveContainer" containerID="67240074f676e6978302775a10add39a0f4dfad391d896bcd478f5ff53394643"
Nov 03 19:24:22.081988 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:22.081730    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"
Nov 03 19:24:22.264504 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:22.264471    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"]
Nov 03 19:24:22.272701 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:22.272585    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-942af8/test-model-942af8-kserve-6bb8f79d4c-z7zh8"]
Nov 03 19:24:22.993458 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:24:22.993427    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" path="/var/lib/kubelet/pods/fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b/volumes"
Nov 03 19:24:47.530252 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:24:47.530211    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:25:09.408360 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:09.408273    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"]
Nov 03 19:25:09.408906 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:09.408485    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" containerID="cri-o://c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08" gracePeriod=30
Nov 03 19:25:12.328792 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:12.328759    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-m4ph9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:25:12.329199 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:12.328828    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:25:14.661751 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.661727    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:25:14.813706 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813626    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-data\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.813706 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813699    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-credential-socket\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.813961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813754    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-token\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-token\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.813961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813846    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-envoy\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.813961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813897    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-certs\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.813961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813946    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-data" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:14.813961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813958    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-kube-api-access-wtllk\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-kube-api-access-wtllk\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.814253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.813996    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istiod-ca-cert\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.814253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814016    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-socket\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.814253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814010    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-credential-socket" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:14.814253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814077    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-podinfo\") pod \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\" (UID: \"0683b2c2-7569-4ab3-80d9-340c7b50a7ab\") "
Nov 03 19:25:14.814253 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814228    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-certs" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:14.814489 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814269    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-socket" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:14.814489 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814383    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istiod-ca-cert" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:14.814585 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814565    4489 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.814624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814593    4489 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istiod-ca-cert\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.814624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814607    4489 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-workload-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.814716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814622    4489 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-data\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.814716 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.814636    4489 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-credential-socket\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.816350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.816325    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-envoy" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:14.816350 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.816337    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-podinfo" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:25:14.816495 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.816347    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-token" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:14.816495 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.816358    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-kube-api-access-wtllk" pod "0683b2c2-7569-4ab3-80d9-340c7b50a7ab" (UID: "0683b2c2-7569-4ab3-80d9-340c7b50a7ab"). InnerVolumeSpecName "kube-api-access-wtllk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:14.916079 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.916044    4489 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-podinfo\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.916079 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.916074    4489 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-token\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.916079 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.916084    4489 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-istio-envoy\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:14.916321 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:14.916093    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtllk\" (UniqueName: \"kubernetes.io/projected/0683b2c2-7569-4ab3-80d9-340c7b50a7ab-kube-api-access-wtllk\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:15.341576 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.341540    4489 generic.go:356] "Generic (PLEG): container finished" podID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerID="c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08" exitCode=0
Nov 03 19:25:15.341749 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.341601    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"
Nov 03 19:25:15.341749 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.341599    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" event={"ID":"0683b2c2-7569-4ab3-80d9-340c7b50a7ab","Type":"ContainerDied","Data":"c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08"}
Nov 03 19:25:15.341749 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.341663    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9" event={"ID":"0683b2c2-7569-4ab3-80d9-340c7b50a7ab","Type":"ContainerDied","Data":"6ef6ffc5a1ed27743d86aee27f5af8ceba8a79983a180600cfd361c147ffdf8e"}
Nov 03 19:25:15.341749 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.341687    4489 scope.go:122] "RemoveContainer" containerID="c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08"
Nov 03 19:25:15.352977 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.352946    4489 scope.go:122] "RemoveContainer" containerID="c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08"
Nov 03 19:25:15.353307 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:25:15.353287    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08\": container with ID starting with c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08 not found: ID does not exist" containerID="c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08"
Nov 03 19:25:15.353370 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.353318    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08"} err="failed to get container status \"c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08\": rpc error: code = NotFound desc = could not find container \"c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08\": container with ID starting with c9ebc0ffda870385321af4c6de673b4256bf3fbd2c3b6f9c5c49235887fc7e08 not found: ID does not exist"
Nov 03 19:25:15.565282 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.565249    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"]
Nov 03 19:25:15.569488 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:15.569454    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-m4ph9"]
Nov 03 19:25:16.993907 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:16.993867    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" path="/var/lib/kubelet/pods/0683b2c2-7569-4ab3-80d9-340c7b50a7ab/volumes"
Nov 03 19:25:17.534916 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:25:17.534862    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:25:23.249575 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:23.249541    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"]
Nov 03 19:25:23.301494 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:23.301457    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"]
Nov 03 19:25:34.973335 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:34.973305    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"]
Nov 03 19:25:34.973788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:34.973528    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n" podUID="2685be93-62a5-4ea2-a56d-5322a23e6113" containerName="maas-api" containerID="cri-o://d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121" gracePeriod=30
Nov 03 19:25:35.239171 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.239095    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:25:35.319343 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.319318    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2685be93-62a5-4ea2-a56d-5322a23e6113-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685be93-62a5-4ea2-a56d-5322a23e6113-openshift-service-ca\") pod \"2685be93-62a5-4ea2-a56d-5322a23e6113\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") "
Nov 03 19:25:35.319475 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.319351    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2685be93-62a5-4ea2-a56d-5322a23e6113-kube-api-access-vzwlw\" (UniqueName: \"kubernetes.io/projected/2685be93-62a5-4ea2-a56d-5322a23e6113-kube-api-access-vzwlw\") pod \"2685be93-62a5-4ea2-a56d-5322a23e6113\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") "
Nov 03 19:25:35.319475 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.319374    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-metrics-certs\") pod \"2685be93-62a5-4ea2-a56d-5322a23e6113\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") "
Nov 03 19:25:35.319475 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.319391    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls\") pod \"2685be93-62a5-4ea2-a56d-5322a23e6113\" (UID: \"2685be93-62a5-4ea2-a56d-5322a23e6113\") "
Nov 03 19:25:35.319689 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.319664    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2685be93-62a5-4ea2-a56d-5322a23e6113-openshift-service-ca" pod "2685be93-62a5-4ea2-a56d-5322a23e6113" (UID: "2685be93-62a5-4ea2-a56d-5322a23e6113"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:35.321632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.321604    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-metrics-certs" pod "2685be93-62a5-4ea2-a56d-5322a23e6113" (UID: "2685be93-62a5-4ea2-a56d-5322a23e6113"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:25:35.321778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.321757    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2685be93-62a5-4ea2-a56d-5322a23e6113-kube-api-access-vzwlw" pod "2685be93-62a5-4ea2-a56d-5322a23e6113" (UID: "2685be93-62a5-4ea2-a56d-5322a23e6113"). InnerVolumeSpecName "kube-api-access-vzwlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:35.321778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.321771    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls" pod "2685be93-62a5-4ea2-a56d-5322a23e6113" (UID: "2685be93-62a5-4ea2-a56d-5322a23e6113"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:25:35.420303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.420272    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2685be93-62a5-4ea2-a56d-5322a23e6113-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:35.420303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.420298    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwlw\" (UniqueName: \"kubernetes.io/projected/2685be93-62a5-4ea2-a56d-5322a23e6113-kube-api-access-vzwlw\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:35.420303 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.420310    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:35.420544 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.420319    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2685be93-62a5-4ea2-a56d-5322a23e6113-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:35.455994 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.455965    4489 generic.go:356] "Generic (PLEG): container finished" podID="2685be93-62a5-4ea2-a56d-5322a23e6113" containerID="d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121" exitCode=0
Nov 03 19:25:35.456160 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.456010    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"
Nov 03 19:25:35.456160 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.456049    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n" event={"ID":"2685be93-62a5-4ea2-a56d-5322a23e6113","Type":"ContainerDied","Data":"d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121"}
Nov 03 19:25:35.456160 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.456107    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n" event={"ID":"2685be93-62a5-4ea2-a56d-5322a23e6113","Type":"ContainerDied","Data":"49d15502977b3261edcaeec03ccae38321171c0502b48111a6e5b7cc32ab76cc"}
Nov 03 19:25:35.456160 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.456137    4489 scope.go:122] "RemoveContainer" containerID="d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121"
Nov 03 19:25:35.467166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.466903    4489 scope.go:122] "RemoveContainer" containerID="d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121"
Nov 03 19:25:35.467458 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:25:35.467360    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121\": container with ID starting with d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121 not found: ID does not exist" containerID="d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121"
Nov 03 19:25:35.467458 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.467395    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121"} err="failed to get container status \"d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121\": rpc error: code = NotFound desc = could not find container \"d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121\": container with ID starting with d3203982843189e76cbc87d454682565c0cb6df69a99aa3932e960c3a1c37121 not found: ID does not exist"
Nov 03 19:25:35.627646 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.627561    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"]
Nov 03 19:25:35.636803 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:35.636779    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770e67-db85f5ccb-f942n"]
Nov 03 19:25:36.993737 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:36.993707    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2685be93-62a5-4ea2-a56d-5322a23e6113" path="/var/lib/kubelet/pods/2685be93-62a5-4ea2-a56d-5322a23e6113/volumes"
Nov 03 19:25:38.270109 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.270050    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main" containerID="cri-o://cf1c3442fdb7d9eff2f20c4c11c8c324b442a646b3c18c435aefdf29ec5b3fdb" gracePeriod=45
Nov 03 19:25:38.321096 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.321066    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" containerID="cri-o://38a36ac2cb4e7b108bb7bf7a05e74b49b63b79c551e98985d1089701f1be6b10" gracePeriod=45
Nov 03 19:25:38.435166 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.435118    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused"
Nov 03 19:25:38.472843 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.472812    4489 generic.go:356] "Generic (PLEG): container finished" podID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerID="cf1c3442fdb7d9eff2f20c4c11c8c324b442a646b3c18c435aefdf29ec5b3fdb" exitCode=0
Nov 03 19:25:38.472995 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.472882    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" event={"ID":"7e3760b0-efff-4d4d-96f9-f834a1812dca","Type":"ContainerDied","Data":"cf1c3442fdb7d9eff2f20c4c11c8c324b442a646b3c18c435aefdf29ec5b3fdb"}
Nov 03 19:25:38.474486 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.474462    4489 generic.go:356] "Generic (PLEG): container finished" podID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerID="38a36ac2cb4e7b108bb7bf7a05e74b49b63b79c551e98985d1089701f1be6b10" exitCode=0
Nov 03 19:25:38.474629 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.474500    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" event={"ID":"565d86a6-1df3-4a4d-aefd-44369e76b11e","Type":"ContainerDied","Data":"38a36ac2cb4e7b108bb7bf7a05e74b49b63b79c551e98985d1089701f1be6b10"}
Nov 03 19:25:38.571270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.571245    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:25:38.596216 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.596193    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:25:38.650942 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.650918    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3760b0-efff-4d4d-96f9-f834a1812dca-kube-api-access-zh5vv\" (UniqueName: \"kubernetes.io/projected/7e3760b0-efff-4d4d-96f9-f834a1812dca-kube-api-access-zh5vv\") pod \"7e3760b0-efff-4d4d-96f9-f834a1812dca\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") "
Nov 03 19:25:38.651079 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.650979    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-model-cache\") pod \"7e3760b0-efff-4d4d-96f9-f834a1812dca\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") "
Nov 03 19:25:38.651079 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651008    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/565d86a6-1df3-4a4d-aefd-44369e76b11e-tls-certs\" (UniqueName: \"kubernetes.io/secret/565d86a6-1df3-4a4d-aefd-44369e76b11e-tls-certs\") pod \"565d86a6-1df3-4a4d-aefd-44369e76b11e\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") "
Nov 03 19:25:38.651079 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651028    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e3760b0-efff-4d4d-96f9-f834a1812dca-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3760b0-efff-4d4d-96f9-f834a1812dca-tls-certs\") pod \"7e3760b0-efff-4d4d-96f9-f834a1812dca\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") "
Nov 03 19:25:38.651292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651101    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565d86a6-1df3-4a4d-aefd-44369e76b11e-kube-api-access-f8j6w\" (UniqueName: \"kubernetes.io/projected/565d86a6-1df3-4a4d-aefd-44369e76b11e-kube-api-access-f8j6w\") pod \"565d86a6-1df3-4a4d-aefd-44369e76b11e\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") "
Nov 03 19:25:38.651292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651180    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-model-cache\") pod \"565d86a6-1df3-4a4d-aefd-44369e76b11e\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") "
Nov 03 19:25:38.651292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651204    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-dshm\") pod \"565d86a6-1df3-4a4d-aefd-44369e76b11e\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") "
Nov 03 19:25:38.651292 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651237    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-home\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-home\") pod \"565d86a6-1df3-4a4d-aefd-44369e76b11e\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") "
Nov 03 19:25:38.651486 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651331    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-model-cache" pod "7e3760b0-efff-4d4d-96f9-f834a1812dca" (UID: "7e3760b0-efff-4d4d-96f9-f834a1812dca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.651547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651503    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-model-cache" pod "565d86a6-1df3-4a4d-aefd-44369e76b11e" (UID: "565d86a6-1df3-4a4d-aefd-44369e76b11e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.651603 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651591    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-dshm\") pod \"7e3760b0-efff-4d4d-96f9-f834a1812dca\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") "
Nov 03 19:25:38.651660 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651624    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-home" pod "565d86a6-1df3-4a4d-aefd-44369e76b11e" (UID: "565d86a6-1df3-4a4d-aefd-44369e76b11e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.651660 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651640    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-tmp-dir\") pod \"565d86a6-1df3-4a4d-aefd-44369e76b11e\" (UID: \"565d86a6-1df3-4a4d-aefd-44369e76b11e\") "
Nov 03 19:25:38.651776 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651674    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-home\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-home\") pod \"7e3760b0-efff-4d4d-96f9-f834a1812dca\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") "
Nov 03 19:25:38.651776 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651708    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-tmp-dir\") pod \"7e3760b0-efff-4d4d-96f9-f834a1812dca\" (UID: \"7e3760b0-efff-4d4d-96f9-f834a1812dca\") "
Nov 03 19:25:38.651961 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.651936    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-tmp-dir" pod "565d86a6-1df3-4a4d-aefd-44369e76b11e" (UID: "565d86a6-1df3-4a4d-aefd-44369e76b11e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.652205 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.652130    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-home" pod "7e3760b0-efff-4d4d-96f9-f834a1812dca" (UID: "7e3760b0-efff-4d4d-96f9-f834a1812dca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.652411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.652337    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.652411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.652357    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.652411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.652371    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.652411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.652379    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-tmp-dir" pod "7e3760b0-efff-4d4d-96f9-f834a1812dca" (UID: "7e3760b0-efff-4d4d-96f9-f834a1812dca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.652411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.652384    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.652730 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.652428    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.653740 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.653714    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3760b0-efff-4d4d-96f9-f834a1812dca-tls-certs" pod "7e3760b0-efff-4d4d-96f9-f834a1812dca" (UID: "7e3760b0-efff-4d4d-96f9-f834a1812dca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:25:38.653740 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.653733    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3760b0-efff-4d4d-96f9-f834a1812dca-kube-api-access-zh5vv" pod "7e3760b0-efff-4d4d-96f9-f834a1812dca" (UID: "7e3760b0-efff-4d4d-96f9-f834a1812dca"). InnerVolumeSpecName "kube-api-access-zh5vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:38.653959 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.653942    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565d86a6-1df3-4a4d-aefd-44369e76b11e-kube-api-access-f8j6w" pod "565d86a6-1df3-4a4d-aefd-44369e76b11e" (UID: "565d86a6-1df3-4a4d-aefd-44369e76b11e"). InnerVolumeSpecName "kube-api-access-f8j6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:38.654018 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.653952    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565d86a6-1df3-4a4d-aefd-44369e76b11e-tls-certs" pod "565d86a6-1df3-4a4d-aefd-44369e76b11e" (UID: "565d86a6-1df3-4a4d-aefd-44369e76b11e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:25:38.654058 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.654013    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-dshm" pod "7e3760b0-efff-4d4d-96f9-f834a1812dca" (UID: "7e3760b0-efff-4d4d-96f9-f834a1812dca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.654122 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.654108    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-dshm" pod "565d86a6-1df3-4a4d-aefd-44369e76b11e" (UID: "565d86a6-1df3-4a4d-aefd-44369e76b11e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:38.753194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.753162    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8j6w\" (UniqueName: \"kubernetes.io/projected/565d86a6-1df3-4a4d-aefd-44369e76b11e-kube-api-access-f8j6w\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.753194 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.753193    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565d86a6-1df3-4a4d-aefd-44369e76b11e-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.753386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.753204    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.753386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.753212    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3760b0-efff-4d4d-96f9-f834a1812dca-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.753386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.753220    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh5vv\" (UniqueName: \"kubernetes.io/projected/7e3760b0-efff-4d4d-96f9-f834a1812dca-kube-api-access-zh5vv\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.753386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.753228    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565d86a6-1df3-4a4d-aefd-44369e76b11e-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:38.753386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:38.753237    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e3760b0-efff-4d4d-96f9-f834a1812dca-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:39.482067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.482033    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"
Nov 03 19:25:39.482067 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.482040    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv" event={"ID":"565d86a6-1df3-4a4d-aefd-44369e76b11e","Type":"ContainerDied","Data":"ed9a3ab18d2a8a7d119f02e06a8b4674170412e9e98fb357844f730050d637ff"}
Nov 03 19:25:39.482631 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.482093    4489 scope.go:122] "RemoveContainer" containerID="38a36ac2cb4e7b108bb7bf7a05e74b49b63b79c551e98985d1089701f1be6b10"
Nov 03 19:25:39.483451 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.483426    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6" event={"ID":"7e3760b0-efff-4d4d-96f9-f834a1812dca","Type":"ContainerDied","Data":"a3ed318473ac64662ffe5847b730c4f36c439199268d8b5a90d12ccf44129460"}
Nov 03 19:25:39.483514 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.483484    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"
Nov 03 19:25:39.491165 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.491124    4489 scope.go:122] "RemoveContainer" containerID="cf1c3442fdb7d9eff2f20c4c11c8c324b442a646b3c18c435aefdf29ec5b3fdb"
Nov 03 19:25:39.652747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.652653    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"]
Nov 03 19:25:39.654822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.654797    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/simulator-w1-355e1a-kserve-77dd4476-s46jv"]
Nov 03 19:25:39.768220 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.768182    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"]
Nov 03 19:25:39.772467 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:39.772443    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-2-w1-355e1a-kserve-684b8cb8c9-6bpj6"]
Nov 03 19:25:40.993883 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:40.993848    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" path="/var/lib/kubelet/pods/565d86a6-1df3-4a4d-aefd-44369e76b11e/volumes"
Nov 03 19:25:40.994308 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:25:40.994292    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" path="/var/lib/kubelet/pods/7e3760b0-efff-4d4d-96f9-f834a1812dca/volumes"
Nov 03 19:25:47.530804 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:25:47.530772    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:26:00.932326 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:00.932296    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"]
Nov 03 19:26:00.932836 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:00.932517    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr" podUID="d4d6305a-25c1-4534-aaec-0dc76184c358" containerName="maas-api" containerID="cri-o://ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc" gracePeriod=30
Nov 03 19:26:01.186656 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.186598    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 19:26:01.374221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.374188    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d6305a-25c1-4534-aaec-0dc76184c358-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6305a-25c1-4534-aaec-0dc76184c358-openshift-service-ca\") pod \"d4d6305a-25c1-4534-aaec-0dc76184c358\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") "
Nov 03 19:26:01.374221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.374229    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d6305a-25c1-4534-aaec-0dc76184c358-kube-api-access-zjhrz\" (UniqueName: \"kubernetes.io/projected/d4d6305a-25c1-4534-aaec-0dc76184c358-kube-api-access-zjhrz\") pod \"d4d6305a-25c1-4534-aaec-0dc76184c358\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") "
Nov 03 19:26:01.374422 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.374280    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls\") pod \"d4d6305a-25c1-4534-aaec-0dc76184c358\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") "
Nov 03 19:26:01.374422 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.374369    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-metrics-certs\") pod \"d4d6305a-25c1-4534-aaec-0dc76184c358\" (UID: \"d4d6305a-25c1-4534-aaec-0dc76184c358\") "
Nov 03 19:26:01.374590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.374564    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d6305a-25c1-4534-aaec-0dc76184c358-openshift-service-ca" pod "d4d6305a-25c1-4534-aaec-0dc76184c358" (UID: "d4d6305a-25c1-4534-aaec-0dc76184c358"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:26:01.376550 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.376516    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-metrics-certs" pod "d4d6305a-25c1-4534-aaec-0dc76184c358" (UID: "d4d6305a-25c1-4534-aaec-0dc76184c358"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:01.376550 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.376531    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls" pod "d4d6305a-25c1-4534-aaec-0dc76184c358" (UID: "d4d6305a-25c1-4534-aaec-0dc76184c358"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:01.376717 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.376597    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d6305a-25c1-4534-aaec-0dc76184c358-kube-api-access-zjhrz" pod "d4d6305a-25c1-4534-aaec-0dc76184c358" (UID: "d4d6305a-25c1-4534-aaec-0dc76184c358"). InnerVolumeSpecName "kube-api-access-zjhrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:01.476127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.476044    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d6305a-25c1-4534-aaec-0dc76184c358-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:01.476127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.476069    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhrz\" (UniqueName: \"kubernetes.io/projected/d4d6305a-25c1-4534-aaec-0dc76184c358-kube-api-access-zjhrz\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:01.476127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.476080    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:01.476127 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.476091    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d6305a-25c1-4534-aaec-0dc76184c358-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:01.588459 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.588421    4489 generic.go:356] "Generic (PLEG): container finished" podID="d4d6305a-25c1-4534-aaec-0dc76184c358" containerID="ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc" exitCode=0
Nov 03 19:26:01.588622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.588474    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"
Nov 03 19:26:01.588622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.588465    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr" event={"ID":"d4d6305a-25c1-4534-aaec-0dc76184c358","Type":"ContainerDied","Data":"ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc"}
Nov 03 19:26:01.588622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.588585    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr" event={"ID":"d4d6305a-25c1-4534-aaec-0dc76184c358","Type":"ContainerDied","Data":"f3a220817521fbf6529e80d2f5b058612920412b00deb65f0fafabc3b302b65c"}
Nov 03 19:26:01.588622 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.588611    4489 scope.go:122] "RemoveContainer" containerID="ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc"
Nov 03 19:26:01.598345 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.598329    4489 scope.go:122] "RemoveContainer" containerID="ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc"
Nov 03 19:26:01.598612 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:26:01.598595    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc\": container with ID starting with ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc not found: ID does not exist" containerID="ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc"
Nov 03 19:26:01.598660 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.598622    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc"} err="failed to get container status \"ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc\": rpc error: code = NotFound desc = could not find container \"ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc\": container with ID starting with ae479b69ee4ddff1b07ce2c3d913ea370ff589f3582ef48238eef1f322b013cc not found: ID does not exist"
Nov 03 19:26:01.758573 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.758498    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"]
Nov 03 19:26:01.762430 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:01.762404    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-355e1a-785bf59f9b-2zmlr"]
Nov 03 19:26:02.993824 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:02.993789    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d6305a-25c1-4534-aaec-0dc76184c358" path="/var/lib/kubelet/pods/d4d6305a-25c1-4534-aaec-0dc76184c358/volumes"
Nov 03 19:26:17.530099 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:26:17.530065    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:26:32.051772 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:32.051682    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"]
Nov 03 19:26:32.103528 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:32.103493    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d24373-s72d4"]
Nov 03 19:26:32.993972 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:32.993930    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed198511-d5dd-4f0c-afd8-672df97752f7" path="/var/lib/kubelet/pods/ed198511-d5dd-4f0c-afd8-672df97752f7/volumes"
Nov 03 19:26:35.769593 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:35.769560    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"]
Nov 03 19:26:47.528948 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:26:47.528912    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:26:50.789218 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:50.789154    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main" containerID="cri-o://61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef" gracePeriod=45
Nov 03 19:26:51.048120 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.048065    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:26:51.133990 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.133956    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d0751b0-b70c-4163-b158-7db0c3c9432e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0751b0-b70c-4163-b158-7db0c3c9432e-tls-certs\") pod \"4d0751b0-b70c-4163-b158-7db0c3c9432e\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") "
Nov 03 19:26:51.134178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134014    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-home\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-home\") pod \"4d0751b0-b70c-4163-b158-7db0c3c9432e\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") "
Nov 03 19:26:51.134178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134056    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-dshm\") pod \"4d0751b0-b70c-4163-b158-7db0c3c9432e\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") "
Nov 03 19:26:51.134178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134101    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-model-cache\") pod \"4d0751b0-b70c-4163-b158-7db0c3c9432e\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") "
Nov 03 19:26:51.134357 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134178    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0751b0-b70c-4163-b158-7db0c3c9432e-kube-api-access-s9ffp\" (UniqueName: \"kubernetes.io/projected/4d0751b0-b70c-4163-b158-7db0c3c9432e-kube-api-access-s9ffp\") pod \"4d0751b0-b70c-4163-b158-7db0c3c9432e\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") "
Nov 03 19:26:51.134357 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134281    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-home" pod "4d0751b0-b70c-4163-b158-7db0c3c9432e" (UID: "4d0751b0-b70c-4163-b158-7db0c3c9432e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:51.134357 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134290    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-tmp-dir\") pod \"4d0751b0-b70c-4163-b158-7db0c3c9432e\" (UID: \"4d0751b0-b70c-4163-b158-7db0c3c9432e\") "
Nov 03 19:26:51.134511 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134369    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-model-cache" pod "4d0751b0-b70c-4163-b158-7db0c3c9432e" (UID: "4d0751b0-b70c-4163-b158-7db0c3c9432e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:51.134511 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134478    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-tmp-dir" pod "4d0751b0-b70c-4163-b158-7db0c3c9432e" (UID: "4d0751b0-b70c-4163-b158-7db0c3c9432e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:51.134628 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134615    4489 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-tmp-dir\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:51.134668 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134632    4489 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-home\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:51.134668 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.134642    4489 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-model-cache\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:51.137270 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.137241    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0751b0-b70c-4163-b158-7db0c3c9432e-tls-certs" pod "4d0751b0-b70c-4163-b158-7db0c3c9432e" (UID: "4d0751b0-b70c-4163-b158-7db0c3c9432e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:51.137677 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.137646    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-dshm" pod "4d0751b0-b70c-4163-b158-7db0c3c9432e" (UID: "4d0751b0-b70c-4163-b158-7db0c3c9432e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:51.140275 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.140230    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0751b0-b70c-4163-b158-7db0c3c9432e-kube-api-access-s9ffp" pod "4d0751b0-b70c-4163-b158-7db0c3c9432e" (UID: "4d0751b0-b70c-4163-b158-7db0c3c9432e"). InnerVolumeSpecName "kube-api-access-s9ffp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:51.235533 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.235504    4489 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d0751b0-b70c-4163-b158-7db0c3c9432e-tls-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:51.235533 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.235530    4489 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d0751b0-b70c-4163-b158-7db0c3c9432e-dshm\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:51.235533 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.235541    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9ffp\" (UniqueName: \"kubernetes.io/projected/4d0751b0-b70c-4163-b158-7db0c3c9432e-kube-api-access-s9ffp\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:51.807468 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.807367    4489 scope.go:122] "RemoveContainer" containerID="61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.813124    4489 generic.go:356] "Generic (PLEG): container finished" podID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerID="61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef" exitCode=0
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.813184    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" event={"ID":"4d0751b0-b70c-4163-b158-7db0c3c9432e","Type":"ContainerDied","Data":"61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"}
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.813246    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps" event={"ID":"4d0751b0-b70c-4163-b158-7db0c3c9432e","Type":"ContainerDied","Data":"ebc4b21852c8a97972c4dc0c786097248f49272760286f0980f7f201eca17b1f"}
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.813265    4489 scope.go:122] "RemoveContainer" containerID="61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.813202    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.816550    4489 scope.go:122] "RemoveContainer" containerID="61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:26:51.816818    4489 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_main_premium-w2-32a59b-kserve-6f48b9c564-t8vps_e2e-models-e2e-worker-w2-32a59b_4d0751b0-b70c-4163-b158-7db0c3c9432e_0 in pod sandbox ebc4b21852c8a97972c4dc0c786097248f49272760286f0980f7f201eca17b1f from index: no such id: '61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef'" containerID="61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:26:51.816853    4489 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_main_premium-w2-32a59b-kserve-6f48b9c564-t8vps_e2e-models-e2e-worker-w2-32a59b_4d0751b0-b70c-4163-b158-7db0c3c9432e_0 in pod sandbox ebc4b21852c8a97972c4dc0c786097248f49272760286f0980f7f201eca17b1f from index: no such id: '61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef'" containerID="61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:26:51.816827    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef\": container with ID starting with 61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef not found: ID does not exist" containerID="61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"
Nov 03 19:26:51.821299 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.816921    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef"} err="failed to get container status \"61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef\": rpc error: code = NotFound desc = could not find container \"61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef\": container with ID starting with 61fe07d689ce4f1c58a28ee1df4251234458cf80e79570052c454310a69d22ef not found: ID does not exist"
Nov 03 19:26:51.981302 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:51.981275    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"]
Nov 03 19:26:52.033249 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:52.033223    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/premium-w2-32a59b-kserve-6f48b9c564-t8vps"]
Nov 03 19:26:52.993571 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:26:52.993543    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" path="/var/lib/kubelet/pods/4d0751b0-b70c-4163-b158-7db0c3c9432e/volumes"
Nov 03 19:27:17.529049 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:27:17.529013    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:27:47.529210 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:27:47.529176    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:27:51.831118 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:27:51.831081    4489 scope.go:122] "RemoveContainer" containerID="a56bbb0ce9ab07e825860f5e44ca9c2f1fbfe2ebe90c09e28184e490a2a050b6"
Nov 03 19:28:17.528951 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:28:17.528874    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:28:47.528756 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:28:47.528714    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:28:54.438175 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438125    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"]
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438467    4489 cpu_manager.go:414] "containerMap: removing container" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438478    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438491    4489 cpu_manager.go:414] "containerMap: removing container" podUID="2a01dcb7-4743-434b-b38e-ef4002c13b3e" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438496    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a01dcb7-4743-434b-b38e-ef4002c13b3e" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438504    4489 cpu_manager.go:414] "containerMap: removing container" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438509    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438521    4489 cpu_manager.go:414] "containerMap: removing container" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438526    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438535    4489 cpu_manager.go:414] "containerMap: removing container" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438540    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438547    4489 cpu_manager.go:414] "containerMap: removing container" podUID="2685be93-62a5-4ea2-a56d-5322a23e6113" containerName="maas-api"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438551    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2685be93-62a5-4ea2-a56d-5322a23e6113" containerName="maas-api"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438558    4489 cpu_manager.go:414] "containerMap: removing container" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438562    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438569    4489 cpu_manager.go:414] "containerMap: removing container" podUID="fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438574    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438585    4489 cpu_manager.go:414] "containerMap: removing container" podUID="d4d6305a-25c1-4534-aaec-0dc76184c358" containerName="maas-api"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438590    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d6305a-25c1-4534-aaec-0dc76184c358" containerName="maas-api"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438599    4489 cpu_manager.go:414] "containerMap: removing container" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438604    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438611    4489 cpu_manager.go:414] "containerMap: removing container" podUID="ed198511-d5dd-4f0c-afd8-672df97752f7" containerName="revoke-keys"
Nov 03 19:28:54.438644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438616    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed198511-d5dd-4f0c-afd8-672df97752f7" containerName="revoke-keys"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438663    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0751b0-b70c-4163-b158-7db0c3c9432e" containerName="main"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438673    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438679    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed198511-d5dd-4f0c-afd8-672df97752f7" containerName="revoke-keys"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438685    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="565d86a6-1df3-4a4d-aefd-44369e76b11e" containerName="main"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438691    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a01dcb7-4743-434b-b38e-ef4002c13b3e" containerName="main"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438696    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="0683b2c2-7569-4ab3-80d9-340c7b50a7ab" containerName="istio-proxy"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438702    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3760b0-efff-4d4d-96f9-f834a1812dca" containerName="main"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438708    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="787654a0-d74e-40dd-9641-db7df5d589cd" containerName="istio-proxy"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438712    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="2685be93-62a5-4ea2-a56d-5322a23e6113" containerName="maas-api"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438716    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0a2408-5a3d-49bc-94f3-eb5d6ea0f74b" containerName="main"
Nov 03 19:28:54.439352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.438722    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d6305a-25c1-4534-aaec-0dc76184c358" containerName="maas-api"
Nov 03 19:28:54.441625 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.441610    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494231 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494196    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494235    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494287    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494334    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494512 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494413    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494512 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494497    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19678a-71fc-4cd3-bc44-085860a485cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494583    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpk2\" (UniqueName: \"kubernetes.io/projected/0f19678a-71fc-4cd3-bc44-085860a485cf-kube-api-access-scpk2\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494671 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494635    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.494735 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.494718    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.549776 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.549735    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"]
Nov 03 19:28:54.595928 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.595898    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19678a-71fc-4cd3-bc44-085860a485cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596084 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.595944    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpk2\" (UniqueName: \"kubernetes.io/projected/0f19678a-71fc-4cd3-bc44-085860a485cf-kube-api-access-scpk2\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596084 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.595981    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596084 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596004    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596084 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596028    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596084 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596055    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596084 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596080    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596128    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596189    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596433    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596609 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596503    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596609 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596557    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596713 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596642    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.596778 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.596759    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f19678a-71fc-4cd3-bc44-085860a485cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.598661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.598634    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.598818 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.598803    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.659419 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.659389    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f19678a-71fc-4cd3-bc44-085860a485cf-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.659550 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.659450    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpk2\" (UniqueName: \"kubernetes.io/projected/0f19678a-71fc-4cd3-bc44-085860a485cf-kube-api-access-scpk2\") pod \"maas-default-gateway-openshift-default-7854949c9-8vt58\" (UID: \"0f19678a-71fc-4cd3-bc44-085860a485cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:54.753760 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:54.753682    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:55.169337 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:28:55.169310    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f19678a_71fc_4cd3_bc44_085860a485cf.slice/crio-049497a7b517c5f4c50b8c786256cd89df6ae36351c0146488495de64d71c478 WatchSource:0}: Error finding container 049497a7b517c5f4c50b8c786256cd89df6ae36351c0146488495de64d71c478: Status 404 returned error can't find the container with id 049497a7b517c5f4c50b8c786256cd89df6ae36351c0146488495de64d71c478
Nov 03 19:28:55.171787 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.171756    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:28:55.171893 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.171823    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:28:55.171893 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.171857    4489 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:28:55.172347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.172266    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"]
Nov 03 19:28:55.336825 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.336791    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" event={"ID":"0f19678a-71fc-4cd3-bc44-085860a485cf","Type":"ContainerStarted","Data":"3692dbe4911d5a982595b5679e89585880bca9626e2e1b38b71108c3431bdea5"}
Nov 03 19:28:55.337016 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.336853    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" event={"ID":"0f19678a-71fc-4cd3-bc44-085860a485cf","Type":"ContainerStarted","Data":"049497a7b517c5f4c50b8c786256cd89df6ae36351c0146488495de64d71c478"}
Nov 03 19:28:55.461572 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.461449    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" podStartSLOduration=1.461431412 podStartE2EDuration="1.461431412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:28:55.460551199 +0000 UTC m=+3064.609245762" watchObservedRunningTime="2026-11-03 19:28:55.461431412 +0000 UTC m=+3064.610125985"
Nov 03 19:28:55.762374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:55.762264    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:28:56.768269 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:56.768230    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8vt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:28:56.768863 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:56.768831    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" podUID="0f19678a-71fc-4cd3-bc44-085860a485cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:28:57.754745 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:57.754708    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8vt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:28:57.755042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:57.755013    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" podUID="0f19678a-71fc-4cd3-bc44-085860a485cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:28:58.762596 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:58.762557    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8vt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:28:58.763248 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:58.763216    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" podUID="0f19678a-71fc-4cd3-bc44-085860a485cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:28:59.762180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:59.761523    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8vt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:28:59.762180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:28:59.761589    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" podUID="0f19678a-71fc-4cd3-bc44-085860a485cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:29:00.761734 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:00.761698    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8vt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:29:00.762370 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:00.762338    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" podUID="0f19678a-71fc-4cd3-bc44-085860a485cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:29:01.754867 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:01.754836    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-8vt58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:29:01.755041 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:01.754893    4489 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58" podUID="0f19678a-71fc-4cd3-bc44-085860a485cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:29:03.329824 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:03.329777    4489 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:29:03.330434 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:03.330270    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:29:03.331232 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:03.331208    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-8vt58"
Nov 03 19:29:13.184296 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:13.184239    4489 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-zlt64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:29:13.184948 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:29:13.184896    4489 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-zlt64" podUID="f837beb4-df93-41a8-ad75-0381fc6c0995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:29:17.529564 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:29:17.529530    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:29:47.528685 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:29:47.528635    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:30:17.528655 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:30:17.528612    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:30:47.529237 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:30:47.529201    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:31:17.529515 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:31:17.529436    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:31:47.529817 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:31:47.529771    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:32:17.530914 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:32:17.530871    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:32:47.528852 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:32:47.528812    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:33:17.529020 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:33:17.528985    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:33:47.529012 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:33:47.528971    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:34:17.528985 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:34:17.528908    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:34:47.581348 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:34:47.581304    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:35:17.531250 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:35:17.531206    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:35:47.530358 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:35:47.530318    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:36:17.529746 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:36:17.529704    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:36:47.529778 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:36:47.529734    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:37:17.530884 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:37:17.530851    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:37:47.528473 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:37:47.528429    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:38:17.528852 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:38:17.528812    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:38:47.529121 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:38:47.529085    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:39:17.528839 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:39:17.528799    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:39:47.529456 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:39:47.529412    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:40:17.529540 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:40:17.529456    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:40:47.528489 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:40:47.528450    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:41:17.528107 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:41:17.528068    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:41:47.528497 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:41:47.528414    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:42:17.529203 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:42:17.529168    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:42:47.529209 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:42:47.529175    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:43:17.529117 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:43:17.529023    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:43:47.528900 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:43:47.528863    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:44:17.531333 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:44:17.531291    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:44:24.487651 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:24.487610    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"]
Nov 03 19:44:24.488222 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:24.487858    4489 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4" podUID="c7f9a639-2fdb-423e-b71d-ec7b0330fa50" containerName="maas-api" containerID="cri-o://be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e" gracePeriod=30
Nov 03 19:44:25.731095 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.731070    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 19:44:25.809038 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.808980    4489 generic.go:356] "Generic (PLEG): container finished" podID="c7f9a639-2fdb-423e-b71d-ec7b0330fa50" containerID="be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e" exitCode=0
Nov 03 19:44:25.809038 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.809025    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"
Nov 03 19:44:25.809038 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.809022    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4" event={"ID":"c7f9a639-2fdb-423e-b71d-ec7b0330fa50","Type":"ContainerDied","Data":"be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e"}
Nov 03 19:44:25.809243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.809076    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4" event={"ID":"c7f9a639-2fdb-423e-b71d-ec7b0330fa50","Type":"ContainerDied","Data":"33bfd9fe7cc021e8d82e48d92b344184ae519b0d7d7f22b2b804f4afb2acc1d5"}
Nov 03 19:44:25.809243 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.809098    4489 scope.go:122] "RemoveContainer" containerID="be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e"
Nov 03 19:44:25.817575 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.817546    4489 scope.go:122] "RemoveContainer" containerID="be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e"
Nov 03 19:44:25.817831 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:44:25.817805    4489 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e\": container with ID starting with be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e not found: ID does not exist" containerID="be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e"
Nov 03 19:44:25.817888 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.817842    4489 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e"} err="failed to get container status \"be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e\": rpc error: code = NotFound desc = could not find container \"be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e\": container with ID starting with be7b8874ff171b62254fb7294187818b0215bb1fb324cbb6a09df75b0b8fa45e not found: ID does not exist"
Nov 03 19:44:25.874783 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.874753    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-metrics-certs\") pod \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") "
Nov 03 19:44:25.874970 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.874794    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-kube-api-access-sdrjf\" (UniqueName: \"kubernetes.io/projected/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-kube-api-access-sdrjf\") pod \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") "
Nov 03 19:44:25.874970 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.874831    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-openshift-service-ca\") pod \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") "
Nov 03 19:44:25.874970 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.874865    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-maas-api-tls\") pod \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\" (UID: \"c7f9a639-2fdb-423e-b71d-ec7b0330fa50\") "
Nov 03 19:44:25.875281 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.875254    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-openshift-service-ca" pod "c7f9a639-2fdb-423e-b71d-ec7b0330fa50" (UID: "c7f9a639-2fdb-423e-b71d-ec7b0330fa50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:25.877088 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.877060    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-kube-api-access-sdrjf" pod "c7f9a639-2fdb-423e-b71d-ec7b0330fa50" (UID: "c7f9a639-2fdb-423e-b71d-ec7b0330fa50"). InnerVolumeSpecName "kube-api-access-sdrjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:25.877267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.877085    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-maas-api-tls" pod "c7f9a639-2fdb-423e-b71d-ec7b0330fa50" (UID: "c7f9a639-2fdb-423e-b71d-ec7b0330fa50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:25.877267 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.877099    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-metrics-certs" pod "c7f9a639-2fdb-423e-b71d-ec7b0330fa50" (UID: "c7f9a639-2fdb-423e-b71d-ec7b0330fa50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:25.976013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.975984    4489 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-metrics-certs\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:25.976013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.976009    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrjf\" (UniqueName: \"kubernetes.io/projected/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-kube-api-access-sdrjf\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:25.976013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.976020    4489 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-openshift-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:25.976258 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:25.976029    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7f9a639-2fdb-423e-b71d-ec7b0330fa50-maas-api-tls\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:26.233631 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:26.233579    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"]
Nov 03 19:44:26.285432 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:26.285401    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b47b974d-bj2h4"]
Nov 03 19:44:26.999702 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:26.999593    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f9a639-2fdb-423e-b71d-ec7b0330fa50" path="/var/lib/kubelet/pods/c7f9a639-2fdb-423e-b71d-ec7b0330fa50/volumes"
Nov 03 19:44:47.529177 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:44:47.529084    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:44:54.786265 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.786229    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"]
Nov 03 19:44:54.786759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.786593    4489 cpu_manager.go:414] "containerMap: removing container" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy"
Nov 03 19:44:54.786759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.786604    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy"
Nov 03 19:44:54.786759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.786632    4489 cpu_manager.go:414] "containerMap: removing container" podUID="c7f9a639-2fdb-423e-b71d-ec7b0330fa50" containerName="maas-api"
Nov 03 19:44:54.786759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.786637    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f9a639-2fdb-423e-b71d-ec7b0330fa50" containerName="maas-api"
Nov 03 19:44:54.786759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.786692    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7e4ba8-cc0a-427b-aea8-cb387ad31a22" containerName="istio-proxy"
Nov 03 19:44:54.786759 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.786704    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f9a639-2fdb-423e-b71d-ec7b0330fa50" containerName="maas-api"
Nov 03 19:44:54.791042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.791026    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.844178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.844137    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"]
Nov 03 19:44:54.848441 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.848411    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.848584 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.848561    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8lf\" (UniqueName: \"kubernetes.io/projected/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-kube-api-access-tz8lf\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.848659 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.848639    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.894493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.894467    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:44:54.894493 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.894483    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:44:54.894629 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.894522    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:44:54.897133 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.897119    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jvcs8\""
Nov 03 19:44:54.949896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.949872    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8lf\" (UniqueName: \"kubernetes.io/projected/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-kube-api-access-tz8lf\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.949992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.949908    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.949992 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.949957    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.950691 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.950674    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:54.952435 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:54.952418    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:55.012970 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:55.012943    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8lf\" (UniqueName: \"kubernetes.io/projected/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-kube-api-access-tz8lf\") pod \"maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:55.103979 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:55.103902    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:55.471201 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:44:55.471166    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8e200b_2346_46b5_8fc4_4cebf0d171f9.slice/crio-7bb5e4b8e00653a6f8c01a5e7defaa8acc2a04fa40bac60a154e32d183037a17 WatchSource:0}: Error finding container 7bb5e4b8e00653a6f8c01a5e7defaa8acc2a04fa40bac60a154e32d183037a17: Status 404 returned error can't find the container with id 7bb5e4b8e00653a6f8c01a5e7defaa8acc2a04fa40bac60a154e32d183037a17
Nov 03 19:44:55.473414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:55.473390    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"]
Nov 03 19:44:55.934015 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:55.933982    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg" event={"ID":"6e8e200b-2346-46b5-8fc4-4cebf0d171f9","Type":"ContainerStarted","Data":"594a78466bb54daa662df4aa7f4202e6f28db7ca36d3ca8bb6d34fbab8d223a6"}
Nov 03 19:44:55.934386 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:55.934036    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg" event={"ID":"6e8e200b-2346-46b5-8fc4-4cebf0d171f9","Type":"ContainerStarted","Data":"7bb5e4b8e00653a6f8c01a5e7defaa8acc2a04fa40bac60a154e32d183037a17"}
Nov 03 19:44:56.053812 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:56.053717    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg" podStartSLOduration=2.053703219 podStartE2EDuration="2.053703219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:56.052465389 +0000 UTC m=+4025.201159971" watchObservedRunningTime="2026-11-03 19:44:56.053703219 +0000 UTC m=+4025.202397794"
Nov 03 19:44:56.939837 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:56.939801    4489 generic.go:356] "Generic (PLEG): container finished" podID="6e8e200b-2346-46b5-8fc4-4cebf0d171f9" containerID="594a78466bb54daa662df4aa7f4202e6f28db7ca36d3ca8bb6d34fbab8d223a6" exitCode=0
Nov 03 19:44:56.940244 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:56.939888    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg" event={"ID":"6e8e200b-2346-46b5-8fc4-4cebf0d171f9","Type":"ContainerDied","Data":"594a78466bb54daa662df4aa7f4202e6f28db7ca36d3ca8bb6d34fbab8d223a6"}
Nov 03 19:44:58.078452 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.078427    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:44:58.178027 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.177997    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-openshift-ca\") pod \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") "
Nov 03 19:44:58.178201 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.178163    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-xks-ca\") pod \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") "
Nov 03 19:44:58.178273 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.178201    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-kube-api-access-tz8lf\" (UniqueName: \"kubernetes.io/projected/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-kube-api-access-tz8lf\") pod \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\" (UID: \"6e8e200b-2346-46b5-8fc4-4cebf0d171f9\") "
Nov 03 19:44:58.178357 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.178336    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-openshift-ca" pod "6e8e200b-2346-46b5-8fc4-4cebf0d171f9" (UID: "6e8e200b-2346-46b5-8fc4-4cebf0d171f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:58.178506 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.178489    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-openshift-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:58.180416 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.180392    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-kube-api-access-tz8lf" pod "6e8e200b-2346-46b5-8fc4-4cebf0d171f9" (UID: "6e8e200b-2346-46b5-8fc4-4cebf0d171f9"). InnerVolumeSpecName "kube-api-access-tz8lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:58.180416 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.180390    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-xks-ca" pod "6e8e200b-2346-46b5-8fc4-4cebf0d171f9" (UID: "6e8e200b-2346-46b5-8fc4-4cebf0d171f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:58.280017 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.279946    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-maas-api-xks-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:58.280017 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.279972    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8lf\" (UniqueName: \"kubernetes.io/projected/6e8e200b-2346-46b5-8fc4-4cebf0d171f9-kube-api-access-tz8lf\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:58.949898 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.949864    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg" event={"ID":"6e8e200b-2346-46b5-8fc4-4cebf0d171f9","Type":"ContainerDied","Data":"7bb5e4b8e00653a6f8c01a5e7defaa8acc2a04fa40bac60a154e32d183037a17"}
Nov 03 19:44:58.950066 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.949937    4489 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bb5e4b8e00653a6f8c01a5e7defaa8acc2a04fa40bac60a154e32d183037a17"
Nov 03 19:44:58.950066 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:44:58.949877    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"
Nov 03 19:45:00.159196 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.159158    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"]
Nov 03 19:45:00.159624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.159552    4489 cpu_manager.go:414] "containerMap: removing container" podUID="6e8e200b-2346-46b5-8fc4-4cebf0d171f9" containerName="revoke-keys"
Nov 03 19:45:00.159624 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.159566    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8e200b-2346-46b5-8fc4-4cebf0d171f9" containerName="revoke-keys"
Nov 03 19:45:00.159719 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.159665    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8e200b-2346-46b5-8fc4-4cebf0d171f9" containerName="revoke-keys"
Nov 03 19:45:00.162592 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.162571    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"]
Nov 03 19:45:00.162713 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.162691    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.165712 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.165692    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.217768 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.217733    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"]
Nov 03 19:45:00.267896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.267861    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jvcs8\""
Nov 03 19:45:00.267896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.267878    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:45:00.268135 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.267921    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:45:00.271677 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.271653    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"]
Nov 03 19:45:00.302369 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.302343    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pr2h\" (UniqueName: \"kubernetes.io/projected/5d76b9f9-e548-4041-a838-3a5f017e4031-kube-api-access-5pr2h\") pod \"maas-api-key-cleanup-29895585-6w7fl\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.302571 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.302374    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4badb1c3-bab4-4c6f-bf00-37e09deda09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.302571 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.302405    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d76b9f9-e548-4041-a838-3a5f017e4031-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-6w7fl\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.302571 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.302514    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7q6\" (UniqueName: \"kubernetes.io/projected/4badb1c3-bab4-4c6f-bf00-37e09deda09a-kube-api-access-2w7q6\") pod \"maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.403133 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.403098    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pr2h\" (UniqueName: \"kubernetes.io/projected/5d76b9f9-e548-4041-a838-3a5f017e4031-kube-api-access-5pr2h\") pod \"maas-api-key-cleanup-29895585-6w7fl\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.403318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.403137    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4badb1c3-bab4-4c6f-bf00-37e09deda09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.403318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.403196    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d76b9f9-e548-4041-a838-3a5f017e4031-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-6w7fl\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.403318 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.403245    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7q6\" (UniqueName: \"kubernetes.io/projected/4badb1c3-bab4-4c6f-bf00-37e09deda09a-kube-api-access-2w7q6\") pod \"maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.403996 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.403972    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d76b9f9-e548-4041-a838-3a5f017e4031-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-6w7fl\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.404044 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.403972    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4badb1c3-bab4-4c6f-bf00-37e09deda09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.466584 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.466506    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7q6\" (UniqueName: \"kubernetes.io/projected/4badb1c3-bab4-4c6f-bf00-37e09deda09a-kube-api-access-2w7q6\") pod \"maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.466584 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.466560    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pr2h\" (UniqueName: \"kubernetes.io/projected/5d76b9f9-e548-4041-a838-3a5f017e4031-kube-api-access-5pr2h\") pod \"maas-api-key-cleanup-29895585-6w7fl\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.476305 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.476283    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:00.481411 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.481395    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:00.850979 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:45:00.850940    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4badb1c3_bab4_4c6f_bf00_37e09deda09a.slice/crio-7e793061a2c414a84d46f6b8900d2bbc5d16404dc6eb5d1964c54c81172ff98b WatchSource:0}: Error finding container 7e793061a2c414a84d46f6b8900d2bbc5d16404dc6eb5d1964c54c81172ff98b: Status 404 returned error can't find the container with id 7e793061a2c414a84d46f6b8900d2bbc5d16404dc6eb5d1964c54c81172ff98b
Nov 03 19:45:00.853931 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.853908    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"]
Nov 03 19:45:00.897358 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:45:00.897333    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d76b9f9_e548_4041_a838_3a5f017e4031.slice/crio-fc653f1227c1a3cfd18d9f156a1346cf8a799499680e01580429b4d73214e7d0 WatchSource:0}: Error finding container fc653f1227c1a3cfd18d9f156a1346cf8a799499680e01580429b4d73214e7d0: Status 404 returned error can't find the container with id fc653f1227c1a3cfd18d9f156a1346cf8a799499680e01580429b4d73214e7d0
Nov 03 19:45:00.900518 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.900497    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"]
Nov 03 19:45:00.958289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.958259    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j" event={"ID":"4badb1c3-bab4-4c6f-bf00-37e09deda09a","Type":"ContainerStarted","Data":"7e793061a2c414a84d46f6b8900d2bbc5d16404dc6eb5d1964c54c81172ff98b"}
Nov 03 19:45:00.959277 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:00.959249    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" event={"ID":"5d76b9f9-e548-4041-a838-3a5f017e4031","Type":"ContainerStarted","Data":"fc653f1227c1a3cfd18d9f156a1346cf8a799499680e01580429b4d73214e7d0"}
Nov 03 19:45:01.965403 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:01.965322    4489 generic.go:356] "Generic (PLEG): container finished" podID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerID="5663a93bbc84695512651e2aa8d51ba1fcdc514818b6e8de901f79de1365fd36" exitCode=60
Nov 03 19:45:01.965771 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:01.965412    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" event={"ID":"5d76b9f9-e548-4041-a838-3a5f017e4031","Type":"ContainerDied","Data":"5663a93bbc84695512651e2aa8d51ba1fcdc514818b6e8de901f79de1365fd36"}
Nov 03 19:45:01.965771 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:01.965622    4489 scope.go:122] "RemoveContainer" containerID="5663a93bbc84695512651e2aa8d51ba1fcdc514818b6e8de901f79de1365fd36"
Nov 03 19:45:01.967073 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:01.967052    4489 generic.go:356] "Generic (PLEG): container finished" podID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerID="1c1b6b860c228df939f0ad91402dd961b022041ea61d747bc45858fcd6790050" exitCode=60
Nov 03 19:45:01.967155 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:01.967123    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j" event={"ID":"4badb1c3-bab4-4c6f-bf00-37e09deda09a","Type":"ContainerDied","Data":"1c1b6b860c228df939f0ad91402dd961b022041ea61d747bc45858fcd6790050"}
Nov 03 19:45:01.967387 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:01.967371    4489 scope.go:122] "RemoveContainer" containerID="1c1b6b860c228df939f0ad91402dd961b022041ea61d747bc45858fcd6790050"
Nov 03 19:45:02.973869 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.973771    4489 generic.go:356] "Generic (PLEG): container finished" podID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerID="e8143df5959906f9e6a72669f2c7155a341a3c3d76ce4e2be81153ebdccb0c82" exitCode=60
Nov 03 19:45:02.973869 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.973839    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j" event={"ID":"4badb1c3-bab4-4c6f-bf00-37e09deda09a","Type":"ContainerDied","Data":"e8143df5959906f9e6a72669f2c7155a341a3c3d76ce4e2be81153ebdccb0c82"}
Nov 03 19:45:02.974356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.973923    4489 scope.go:122] "RemoveContainer" containerID="1c1b6b860c228df939f0ad91402dd961b022041ea61d747bc45858fcd6790050"
Nov 03 19:45:02.974356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.974175    4489 scope.go:122] "RemoveContainer" containerID="e8143df5959906f9e6a72669f2c7155a341a3c3d76ce4e2be81153ebdccb0c82"
Nov 03 19:45:02.974467 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:45:02.974434    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j_odh-ai-gateway-infra(4badb1c3-bab4-4c6f-bf00-37e09deda09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a"
Nov 03 19:45:02.975922 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.975894    4489 generic.go:356] "Generic (PLEG): container finished" podID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerID="66de452d33962a2d8f6a0702b117a9a57e113eac3b82f7ba8f39afbf68662e3c" exitCode=60
Nov 03 19:45:02.976005 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.975950    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" event={"ID":"5d76b9f9-e548-4041-a838-3a5f017e4031","Type":"ContainerDied","Data":"66de452d33962a2d8f6a0702b117a9a57e113eac3b82f7ba8f39afbf68662e3c"}
Nov 03 19:45:02.976178 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.976162    4489 scope.go:122] "RemoveContainer" containerID="66de452d33962a2d8f6a0702b117a9a57e113eac3b82f7ba8f39afbf68662e3c"
Nov 03 19:45:02.976371 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:45:02.976354    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-6w7fl_odh-ai-gateway-infra(5d76b9f9-e548-4041-a838-3a5f017e4031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031"
Nov 03 19:45:02.984868 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:02.984849    4489 scope.go:122] "RemoveContainer" containerID="5663a93bbc84695512651e2aa8d51ba1fcdc514818b6e8de901f79de1365fd36"
Nov 03 19:45:03.982737 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:03.982707    4489 scope.go:122] "RemoveContainer" containerID="66de452d33962a2d8f6a0702b117a9a57e113eac3b82f7ba8f39afbf68662e3c"
Nov 03 19:45:03.983243 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:45:03.982964    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-6w7fl_odh-ai-gateway-infra(5d76b9f9-e548-4041-a838-3a5f017e4031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031"
Nov 03 19:45:03.984290 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:03.984271    4489 scope.go:122] "RemoveContainer" containerID="e8143df5959906f9e6a72669f2c7155a341a3c3d76ce4e2be81153ebdccb0c82"
Nov 03 19:45:03.984467 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:45:03.984451    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j_odh-ai-gateway-infra(4badb1c3-bab4-4c6f-bf00-37e09deda09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a"
Nov 03 19:45:14.829819 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:14.829766    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"]
Nov 03 19:45:14.982311 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:14.982289    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:14.995425 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:14.995405    4489 scope.go:122] "RemoveContainer" containerID="66de452d33962a2d8f6a0702b117a9a57e113eac3b82f7ba8f39afbf68662e3c"
Nov 03 19:45:15.067950 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.067926    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"
Nov 03 19:45:15.068137 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.067919    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j" event={"ID":"4badb1c3-bab4-4c6f-bf00-37e09deda09a","Type":"ContainerDied","Data":"7e793061a2c414a84d46f6b8900d2bbc5d16404dc6eb5d1964c54c81172ff98b"}
Nov 03 19:45:15.068137 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.068064    4489 scope.go:122] "RemoveContainer" containerID="e8143df5959906f9e6a72669f2c7155a341a3c3d76ce4e2be81153ebdccb0c82"
Nov 03 19:45:15.150323 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.149126    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4badb1c3-bab4-4c6f-bf00-37e09deda09a-kube-api-access-2w7q6\" (UniqueName: \"kubernetes.io/projected/4badb1c3-bab4-4c6f-bf00-37e09deda09a-kube-api-access-2w7q6\") pod \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") "
Nov 03 19:45:15.150323 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.149285    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4badb1c3-bab4-4c6f-bf00-37e09deda09a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4badb1c3-bab4-4c6f-bf00-37e09deda09a-maas-api-service-ca\") pod \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\" (UID: \"4badb1c3-bab4-4c6f-bf00-37e09deda09a\") "
Nov 03 19:45:15.150323 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.149995    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4badb1c3-bab4-4c6f-bf00-37e09deda09a-maas-api-service-ca" pod "4badb1c3-bab4-4c6f-bf00-37e09deda09a" (UID: "4badb1c3-bab4-4c6f-bf00-37e09deda09a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:15.167193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.152890    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4badb1c3-bab4-4c6f-bf00-37e09deda09a-kube-api-access-2w7q6" pod "4badb1c3-bab4-4c6f-bf00-37e09deda09a" (UID: "4badb1c3-bab4-4c6f-bf00-37e09deda09a"). InnerVolumeSpecName "kube-api-access-2w7q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:15.251186 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.251123    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7q6\" (UniqueName: \"kubernetes.io/projected/4badb1c3-bab4-4c6f-bf00-37e09deda09a-kube-api-access-2w7q6\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:15.251186 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.251178    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4badb1c3-bab4-4c6f-bf00-37e09deda09a-maas-api-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:15.491996 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.491962    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"]
Nov 03 19:45:15.495263 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:15.495244    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-50babf0d-29895585-7pn8j"]
Nov 03 19:45:16.074723 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.074639    4489 generic.go:356] "Generic (PLEG): container finished" podID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerID="f1c9dd1c02964696354303d6c7e6e15ad186972d6f3da33c1dc44ec721579dce" exitCode=60
Nov 03 19:45:16.074723 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.074685    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" event={"ID":"5d76b9f9-e548-4041-a838-3a5f017e4031","Type":"ContainerDied","Data":"f1c9dd1c02964696354303d6c7e6e15ad186972d6f3da33c1dc44ec721579dce"}
Nov 03 19:45:16.075224 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.074749    4489 scope.go:122] "RemoveContainer" containerID="66de452d33962a2d8f6a0702b117a9a57e113eac3b82f7ba8f39afbf68662e3c"
Nov 03 19:45:16.075224 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.075073    4489 scope.go:122] "RemoveContainer" containerID="f1c9dd1c02964696354303d6c7e6e15ad186972d6f3da33c1dc44ec721579dce"
Nov 03 19:45:16.075374 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:45:16.075355    4489 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-6w7fl_odh-ai-gateway-infra(5d76b9f9-e548-4041-a838-3a5f017e4031)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031"
Nov 03 19:45:16.850297 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.850264    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"]
Nov 03 19:45:16.902057 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.902029    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50babf0d-4bnhg"]
Nov 03 19:45:16.993121 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.993091    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" path="/var/lib/kubelet/pods/4badb1c3-bab4-4c6f-bf00-37e09deda09a/volumes"
Nov 03 19:45:16.993533 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:16.993519    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8e200b-2346-46b5-8fc4-4cebf0d171f9" path="/var/lib/kubelet/pods/6e8e200b-2346-46b5-8fc4-4cebf0d171f9/volumes"
Nov 03 19:45:17.203609 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.203580    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"]
Nov 03 19:45:17.333833 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.333801    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:17.473017 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.472933    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d76b9f9-e548-4041-a838-3a5f017e4031-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d76b9f9-e548-4041-a838-3a5f017e4031-maas-api-service-ca\") pod \"5d76b9f9-e548-4041-a838-3a5f017e4031\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") "
Nov 03 19:45:17.473204 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.473045    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d76b9f9-e548-4041-a838-3a5f017e4031-kube-api-access-5pr2h\" (UniqueName: \"kubernetes.io/projected/5d76b9f9-e548-4041-a838-3a5f017e4031-kube-api-access-5pr2h\") pod \"5d76b9f9-e548-4041-a838-3a5f017e4031\" (UID: \"5d76b9f9-e548-4041-a838-3a5f017e4031\") "
Nov 03 19:45:17.473324 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.473302    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d76b9f9-e548-4041-a838-3a5f017e4031-maas-api-service-ca" pod "5d76b9f9-e548-4041-a838-3a5f017e4031" (UID: "5d76b9f9-e548-4041-a838-3a5f017e4031"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:17.473444 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.473420    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d76b9f9-e548-4041-a838-3a5f017e4031-maas-api-service-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:17.475289 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.475270    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d76b9f9-e548-4041-a838-3a5f017e4031-kube-api-access-5pr2h" pod "5d76b9f9-e548-4041-a838-3a5f017e4031" (UID: "5d76b9f9-e548-4041-a838-3a5f017e4031"). InnerVolumeSpecName "kube-api-access-5pr2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:17.529959 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:45:17.529926    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:45:17.574640 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:17.574607    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pr2h\" (UniqueName: \"kubernetes.io/projected/5d76b9f9-e548-4041-a838-3a5f017e4031-kube-api-access-5pr2h\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:18.085859 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:18.085827    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"
Nov 03 19:45:18.085859 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:18.085850    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl" event={"ID":"5d76b9f9-e548-4041-a838-3a5f017e4031","Type":"ContainerDied","Data":"fc653f1227c1a3cfd18d9f156a1346cf8a799499680e01580429b4d73214e7d0"}
Nov 03 19:45:18.086120 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:18.085916    4489 scope.go:122] "RemoveContainer" containerID="f1c9dd1c02964696354303d6c7e6e15ad186972d6f3da33c1dc44ec721579dce"
Nov 03 19:45:18.255037 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:18.255006    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"]
Nov 03 19:45:18.307094 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:18.307061    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-6w7fl"]
Nov 03 19:45:19.003160 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:45:19.003100    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" path="/var/lib/kubelet/pods/5d76b9f9-e548-4041-a838-3a5f017e4031/volumes"
Nov 03 19:45:47.528471 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:45:47.528435    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:46:17.528631 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:46:17.528554    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:46:47.529092 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:46:47.529057    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:47:12.455502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455467    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"]
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455825    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455837    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455847    4489 cpu_manager.go:414] "containerMap: removing container" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerName="cleanup"
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455852    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerName="cleanup"
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455862    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455866    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455877    4489 cpu_manager.go:414] "containerMap: removing container" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerName="cleanup"
Nov 03 19:47:12.455896 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455881    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerName="cleanup"
Nov 03 19:47:12.456168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455946    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:47:12.456168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455954    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerName="cleanup"
Nov 03 19:47:12.456168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455960    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:47:12.456168 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.455968    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="4badb1c3-bab4-4c6f-bf00-37e09deda09a" containerName="cleanup"
Nov 03 19:47:12.458941 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.458925    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.513597 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.513567    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:47:12.515346 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.515315    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"]
Nov 03 19:47:12.543680 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.543646    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.543820 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.543694    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2ff\" (UniqueName: \"kubernetes.io/projected/21edeb7f-0744-41b6-b486-1c2c191d6841-kube-api-access-vg2ff\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.543876 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.543814    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.565536 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.565513    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:47:12.565639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.565522    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:47:12.565639 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.565545    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jvcs8\""
Nov 03 19:47:12.644967 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.644938    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.645131 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.644978    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2ff\" (UniqueName: \"kubernetes.io/projected/21edeb7f-0744-41b6-b486-1c2c191d6841-kube-api-access-vg2ff\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.645131 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.645016    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.645644 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.645619    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.647417 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.647395    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.708183 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.708114    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2ff\" (UniqueName: \"kubernetes.io/projected/21edeb7f-0744-41b6-b486-1c2c191d6841-kube-api-access-vg2ff\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:12.771391 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:12.771370    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:13.135602 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:47:13.135570    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21edeb7f_0744_41b6_b486_1c2c191d6841.slice/crio-dd096931745ed1ea4efb97c2837ced5ced1113179d30062a722e6277443f1f6e WatchSource:0}: Error finding container dd096931745ed1ea4efb97c2837ced5ced1113179d30062a722e6277443f1f6e: Status 404 returned error can't find the container with id dd096931745ed1ea4efb97c2837ced5ced1113179d30062a722e6277443f1f6e
Nov 03 19:47:13.138276 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:13.138250    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"]
Nov 03 19:47:13.578491 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:13.578456    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl" event={"ID":"21edeb7f-0744-41b6-b486-1c2c191d6841","Type":"ContainerStarted","Data":"e28791d630e41ab40a98c6641a0bde5133140e361bb2d91e83c879e9ead36074"}
Nov 03 19:47:13.578932 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:13.578548    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl" event={"ID":"21edeb7f-0744-41b6-b486-1c2c191d6841","Type":"ContainerStarted","Data":"dd096931745ed1ea4efb97c2837ced5ced1113179d30062a722e6277443f1f6e"}
Nov 03 19:47:13.700705 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:13.700657    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl" podStartSLOduration=1.700645481 podStartE2EDuration="1.700645481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:13.699492712 +0000 UTC m=+4162.848187283" watchObservedRunningTime="2026-11-03 19:47:13.700645481 +0000 UTC m=+4162.849340051"
Nov 03 19:47:14.584066 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:14.584030    4489 generic.go:356] "Generic (PLEG): container finished" podID="21edeb7f-0744-41b6-b486-1c2c191d6841" containerID="e28791d630e41ab40a98c6641a0bde5133140e361bb2d91e83c879e9ead36074" exitCode=0
Nov 03 19:47:14.584480 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:14.584074    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl" event={"ID":"21edeb7f-0744-41b6-b486-1c2c191d6841","Type":"ContainerDied","Data":"e28791d630e41ab40a98c6641a0bde5133140e361bb2d91e83c879e9ead36074"}
Nov 03 19:47:15.719863 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.719840    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:15.877418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.877335    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-xks-ca\") pod \"21edeb7f-0744-41b6-b486-1c2c191d6841\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") "
Nov 03 19:47:15.877418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.877381    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21edeb7f-0744-41b6-b486-1c2c191d6841-kube-api-access-vg2ff\" (UniqueName: \"kubernetes.io/projected/21edeb7f-0744-41b6-b486-1c2c191d6841-kube-api-access-vg2ff\") pod \"21edeb7f-0744-41b6-b486-1c2c191d6841\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") "
Nov 03 19:47:15.877418 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.877415    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-openshift-ca\") pod \"21edeb7f-0744-41b6-b486-1c2c191d6841\" (UID: \"21edeb7f-0744-41b6-b486-1c2c191d6841\") "
Nov 03 19:47:15.877884 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.877851    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-openshift-ca" pod "21edeb7f-0744-41b6-b486-1c2c191d6841" (UID: "21edeb7f-0744-41b6-b486-1c2c191d6841"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:15.879573 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.879550    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21edeb7f-0744-41b6-b486-1c2c191d6841-kube-api-access-vg2ff" pod "21edeb7f-0744-41b6-b486-1c2c191d6841" (UID: "21edeb7f-0744-41b6-b486-1c2c191d6841"). InnerVolumeSpecName "kube-api-access-vg2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:15.879690 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.879674    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-xks-ca" pod "21edeb7f-0744-41b6-b486-1c2c191d6841" (UID: "21edeb7f-0744-41b6-b486-1c2c191d6841"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:15.978554 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.978522    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-openshift-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:15.978554 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.978554    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edeb7f-0744-41b6-b486-1c2c191d6841-maas-api-xks-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:15.978715 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:15.978563    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg2ff\" (UniqueName: \"kubernetes.io/projected/21edeb7f-0744-41b6-b486-1c2c191d6841-kube-api-access-vg2ff\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:16.593821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:16.593774    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl" event={"ID":"21edeb7f-0744-41b6-b486-1c2c191d6841","Type":"ContainerDied","Data":"dd096931745ed1ea4efb97c2837ced5ced1113179d30062a722e6277443f1f6e"}
Nov 03 19:47:16.593989 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:16.593839    4489 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd096931745ed1ea4efb97c2837ced5ced1113179d30062a722e6277443f1f6e"
Nov 03 19:47:16.593989 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:16.593792    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"
Nov 03 19:47:17.529527 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:47:17.529491    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:47:36.725956 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:36.725867    4489 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"]
Nov 03 19:47:36.727470 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:36.727443    4489 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ad43e7-9jpgl"]
Nov 03 19:47:36.993875 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:47:36.993789    4489 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21edeb7f-0744-41b6-b486-1c2c191d6841" path="/var/lib/kubelet/pods/21edeb7f-0744-41b6-b486-1c2c191d6841/volumes"
Nov 03 19:47:47.528740 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:47:47.528706    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:48:13.935472 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.935432    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"]
Nov 03 19:48:13.936056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.935975    4489 cpu_manager.go:414] "containerMap: removing container" podUID="21edeb7f-0744-41b6-b486-1c2c191d6841" containerName="revoke-keys"
Nov 03 19:48:13.936056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.935996    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21edeb7f-0744-41b6-b486-1c2c191d6841" containerName="revoke-keys"
Nov 03 19:48:13.936056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.936018    4489 cpu_manager.go:414] "containerMap: removing container" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:48:13.936056 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.936026    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:48:13.936313 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.936131    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="21edeb7f-0744-41b6-b486-1c2c191d6841" containerName="revoke-keys"
Nov 03 19:48:13.936313 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.936166    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d76b9f9-e548-4041-a838-3a5f017e4031" containerName="cleanup"
Nov 03 19:48:13.939200 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.939179    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:13.970730 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.970704    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxcdq\" (UniqueName: \"kubernetes.io/projected/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-kube-api-access-gxcdq\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:13.970838 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.970753    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:13.970882 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.970866    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:13.993688 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:13.993660    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"]
Nov 03 19:48:14.044193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.044137    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:48:14.044193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.044165    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:48:14.044193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.044171    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:48:14.045673 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.045653    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jvcs8\""
Nov 03 19:48:14.072400 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.072367    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxcdq\" (UniqueName: \"kubernetes.io/projected/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-kube-api-access-gxcdq\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:14.072579 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.072422    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:14.072579 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.072485    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:14.073097 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.073075    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:14.075088 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.075061    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:14.135569 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.135535    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxcdq\" (UniqueName: \"kubernetes.io/projected/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-kube-api-access-gxcdq\") pod \"maas-api-revoke-sub-f03629db-qjlng\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:14.250789 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.250720    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:14.620527 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:48:14.620496    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8116cdb1_0ed5_477b_91c5_e7a38e6667e8.slice/crio-1a42ceeaeee548acee9b5cec1375c118733262a48bc171e4b7ca2ce19263d99a WatchSource:0}: Error finding container 1a42ceeaeee548acee9b5cec1375c118733262a48bc171e4b7ca2ce19263d99a: Status 404 returned error can't find the container with id 1a42ceeaeee548acee9b5cec1375c118733262a48bc171e4b7ca2ce19263d99a
Nov 03 19:48:14.622827 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.622805    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"]
Nov 03 19:48:14.856260 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:14.856225    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng" event={"ID":"8116cdb1-0ed5-477b-91c5-e7a38e6667e8","Type":"ContainerStarted","Data":"1a42ceeaeee548acee9b5cec1375c118733262a48bc171e4b7ca2ce19263d99a"}
Nov 03 19:48:15.862560 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:15.862475    4489 generic.go:356] "Generic (PLEG): container finished" podID="8116cdb1-0ed5-477b-91c5-e7a38e6667e8" containerID="65644241561e03188762634f6bba37888c0eca87690cafc12f5fdd210990dea1" exitCode=0
Nov 03 19:48:15.862913 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:15.862555    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng" event={"ID":"8116cdb1-0ed5-477b-91c5-e7a38e6667e8","Type":"ContainerDied","Data":"65644241561e03188762634f6bba37888c0eca87690cafc12f5fdd210990dea1"}
Nov 03 19:48:16.993936 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:16.993909    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:17.102013 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.101985    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-xks-ca\") pod \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") "
Nov 03 19:48:17.102193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.102114    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-openshift-ca\") pod \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") "
Nov 03 19:48:17.102193 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.102184    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-kube-api-access-gxcdq\" (UniqueName: \"kubernetes.io/projected/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-kube-api-access-gxcdq\") pod \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\" (UID: \"8116cdb1-0ed5-477b-91c5-e7a38e6667e8\") "
Nov 03 19:48:17.102524 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.102497    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-openshift-ca" pod "8116cdb1-0ed5-477b-91c5-e7a38e6667e8" (UID: "8116cdb1-0ed5-477b-91c5-e7a38e6667e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:17.104407 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.104381    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-xks-ca" pod "8116cdb1-0ed5-477b-91c5-e7a38e6667e8" (UID: "8116cdb1-0ed5-477b-91c5-e7a38e6667e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:17.104502 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.104464    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-kube-api-access-gxcdq" pod "8116cdb1-0ed5-477b-91c5-e7a38e6667e8" (UID: "8116cdb1-0ed5-477b-91c5-e7a38e6667e8"). InnerVolumeSpecName "kube-api-access-gxcdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:17.203175 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.203131    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-openshift-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:17.203175 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.203171    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxcdq\" (UniqueName: \"kubernetes.io/projected/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-kube-api-access-gxcdq\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:17.203356 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.203182    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8116cdb1-0ed5-477b-91c5-e7a38e6667e8-maas-api-xks-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:17.528423 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:48:17.528335    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:48:17.872048 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.871966    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng" event={"ID":"8116cdb1-0ed5-477b-91c5-e7a38e6667e8","Type":"ContainerDied","Data":"1a42ceeaeee548acee9b5cec1375c118733262a48bc171e4b7ca2ce19263d99a"}
Nov 03 19:48:17.872048 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.872010    4489 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a42ceeaeee548acee9b5cec1375c118733262a48bc171e4b7ca2ce19263d99a"
Nov 03 19:48:17.872048 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:17.871973    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qjlng"
Nov 03 19:48:35.044132 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.044102    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"]
Nov 03 19:48:35.044627 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.044465    4489 cpu_manager.go:414] "containerMap: removing container" podUID="8116cdb1-0ed5-477b-91c5-e7a38e6667e8" containerName="revoke-keys"
Nov 03 19:48:35.044627 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.044477    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8116cdb1-0ed5-477b-91c5-e7a38e6667e8" containerName="revoke-keys"
Nov 03 19:48:35.044627 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.044533    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="8116cdb1-0ed5-477b-91c5-e7a38e6667e8" containerName="revoke-keys"
Nov 03 19:48:35.047547 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.047530    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.103347 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.103318    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"]
Nov 03 19:48:35.153054 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.153028    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:48:35.153054 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.153026    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:48:35.153319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.153069    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jvcs8\""
Nov 03 19:48:35.153319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.153069    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:48:35.206182 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.206132    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f9c\" (UniqueName: \"kubernetes.io/projected/719978d4-84a9-4431-b3f4-e06c34c71ddb-kube-api-access-r4f9c\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.206326 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.206268    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.206326 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.206318    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.307129 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.307052    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f9c\" (UniqueName: \"kubernetes.io/projected/719978d4-84a9-4431-b3f4-e06c34c71ddb-kube-api-access-r4f9c\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.307311 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.307135    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.307311 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.307224    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.307852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.307833    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.309476 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.309456    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.370319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.370294    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f9c\" (UniqueName: \"kubernetes.io/projected/719978d4-84a9-4431-b3f4-e06c34c71ddb-kube-api-access-r4f9c\") pod \"maas-api-revoke-sub-0165d7d2-hjx7x\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:35.658969 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:35.658942    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:36.023961 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:48:36.023923    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719978d4_84a9_4431_b3f4_e06c34c71ddb.slice/crio-d6f68ae88b9e7c9a81d18bf6ab5aef2ff582381a193f28aa0b26b01c08c8975f WatchSource:0}: Error finding container d6f68ae88b9e7c9a81d18bf6ab5aef2ff582381a193f28aa0b26b01c08c8975f: Status 404 returned error can't find the container with id d6f68ae88b9e7c9a81d18bf6ab5aef2ff582381a193f28aa0b26b01c08c8975f
Nov 03 19:48:36.028763 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:36.028735    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"]
Nov 03 19:48:36.954783 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:36.954718    4489 generic.go:356] "Generic (PLEG): container finished" podID="719978d4-84a9-4431-b3f4-e06c34c71ddb" containerID="0a7366634cf10ee0bf279cbda65abaa0cd45743095b3b78b2a0cc4e1f102bf01" exitCode=0
Nov 03 19:48:36.954783 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:36.954751    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x" event={"ID":"719978d4-84a9-4431-b3f4-e06c34c71ddb","Type":"ContainerDied","Data":"0a7366634cf10ee0bf279cbda65abaa0cd45743095b3b78b2a0cc4e1f102bf01"}
Nov 03 19:48:36.955218 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:36.954794    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x" event={"ID":"719978d4-84a9-4431-b3f4-e06c34c71ddb","Type":"ContainerStarted","Data":"d6f68ae88b9e7c9a81d18bf6ab5aef2ff582381a193f28aa0b26b01c08c8975f"}
Nov 03 19:48:38.087997 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.087973    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:38.245394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.234058    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719978d4-84a9-4431-b3f4-e06c34c71ddb-kube-api-access-r4f9c\" (UniqueName: \"kubernetes.io/projected/719978d4-84a9-4431-b3f4-e06c34c71ddb-kube-api-access-r4f9c\") pod \"719978d4-84a9-4431-b3f4-e06c34c71ddb\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") "
Nov 03 19:48:38.245394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.234581    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-openshift-ca\") pod \"719978d4-84a9-4431-b3f4-e06c34c71ddb\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") "
Nov 03 19:48:38.245394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.234934    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-openshift-ca" pod "719978d4-84a9-4431-b3f4-e06c34c71ddb" (UID: "719978d4-84a9-4431-b3f4-e06c34c71ddb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:38.245394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.235001    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-xks-ca\") pod \"719978d4-84a9-4431-b3f4-e06c34c71ddb\" (UID: \"719978d4-84a9-4431-b3f4-e06c34c71ddb\") "
Nov 03 19:48:38.245394 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.235424    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-openshift-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:38.245865 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.245834    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-xks-ca" pod "719978d4-84a9-4431-b3f4-e06c34c71ddb" (UID: "719978d4-84a9-4431-b3f4-e06c34c71ddb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:38.245990 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.245959    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719978d4-84a9-4431-b3f4-e06c34c71ddb-kube-api-access-r4f9c" pod "719978d4-84a9-4431-b3f4-e06c34c71ddb" (UID: "719978d4-84a9-4431-b3f4-e06c34c71ddb"). InnerVolumeSpecName "kube-api-access-r4f9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:38.336100 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.336076    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f9c\" (UniqueName: \"kubernetes.io/projected/719978d4-84a9-4431-b3f4-e06c34c71ddb-kube-api-access-r4f9c\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:38.336100 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.336100    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719978d4-84a9-4431-b3f4-e06c34c71ddb-maas-api-xks-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:38.964334 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.964307    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x"
Nov 03 19:48:38.964482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.964306    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hjx7x" event={"ID":"719978d4-84a9-4431-b3f4-e06c34c71ddb","Type":"ContainerDied","Data":"d6f68ae88b9e7c9a81d18bf6ab5aef2ff582381a193f28aa0b26b01c08c8975f"}
Nov 03 19:48:38.964482 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:38.964427    4489 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f68ae88b9e7c9a81d18bf6ab5aef2ff582381a193f28aa0b26b01c08c8975f"
Nov 03 19:48:46.872384 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.872350    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"]
Nov 03 19:48:46.876473 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.873197    4489 cpu_manager.go:414] "containerMap: removing container" podUID="719978d4-84a9-4431-b3f4-e06c34c71ddb" containerName="revoke-keys"
Nov 03 19:48:46.876473 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.873313    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719978d4-84a9-4431-b3f4-e06c34c71ddb" containerName="revoke-keys"
Nov 03 19:48:46.876473 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.873656    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="719978d4-84a9-4431-b3f4-e06c34c71ddb" containerName="revoke-keys"
Nov 03 19:48:46.879300 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.878103    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:46.915510 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.915480    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89gk\" (UniqueName: \"kubernetes.io/projected/d17d828e-6a5e-4200-8e9f-bc78773e791d-kube-api-access-v89gk\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:46.915661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.915535    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:46.915661 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.915637    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:46.930996 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.930964    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"]
Nov 03 19:48:46.980865 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.980839    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:48:46.981029 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.980890    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:48:46.981029 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.980968    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jvcs8\""
Nov 03 19:48:46.984822 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:46.984799    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:48:47.016735 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.016711    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:47.016875 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.016763    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:47.016875 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.016834    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89gk\" (UniqueName: \"kubernetes.io/projected/d17d828e-6a5e-4200-8e9f-bc78773e791d-kube-api-access-v89gk\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:47.017344 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.017324    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:47.019387 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.019356    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:47.080128 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.080098    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89gk\" (UniqueName: \"kubernetes.io/projected/d17d828e-6a5e-4200-8e9f-bc78773e791d-kube-api-access-v89gk\") pod \"maas-api-revoke-sub-6b19338f-2qfzb\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:47.191994 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.191962    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:47.529501 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:48:47.529422    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:48:47.559663 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:48:47.559627    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17d828e_6a5e_4200_8e9f_bc78773e791d.slice/crio-3d26193a159c97f65860401baac1a285b0efc3588cf32c599c22bd2c988e1917 WatchSource:0}: Error finding container 3d26193a159c97f65860401baac1a285b0efc3588cf32c599c22bd2c988e1917: Status 404 returned error can't find the container with id 3d26193a159c97f65860401baac1a285b0efc3588cf32c599c22bd2c988e1917
Nov 03 19:48:47.561568 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:47.561541    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"]
Nov 03 19:48:48.005840 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:48.005802    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb" event={"ID":"d17d828e-6a5e-4200-8e9f-bc78773e791d","Type":"ContainerStarted","Data":"32642c37b721a17326aabf42588c88195b416f12b218bee9173f7ee247c87c89"}
Nov 03 19:48:48.006246 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:48.005861    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb" event={"ID":"d17d828e-6a5e-4200-8e9f-bc78773e791d","Type":"ContainerStarted","Data":"3d26193a159c97f65860401baac1a285b0efc3588cf32c599c22bd2c988e1917"}
Nov 03 19:48:48.126902 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:48.126808    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb" podStartSLOduration=2.126793814 podStartE2EDuration="2.126793814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:48.126066552 +0000 UTC m=+4257.274761122" watchObservedRunningTime="2026-11-03 19:48:48.126793814 +0000 UTC m=+4257.275488384"
Nov 03 19:48:49.011890 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:49.011855    4489 generic.go:356] "Generic (PLEG): container finished" podID="d17d828e-6a5e-4200-8e9f-bc78773e791d" containerID="32642c37b721a17326aabf42588c88195b416f12b218bee9173f7ee247c87c89" exitCode=0
Nov 03 19:48:49.012319 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:49.011950    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb" event={"ID":"d17d828e-6a5e-4200-8e9f-bc78773e791d","Type":"ContainerDied","Data":"32642c37b721a17326aabf42588c88195b416f12b218bee9173f7ee247c87c89"}
Nov 03 19:48:50.146989 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.146966    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:50.252653 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.252615    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17d828e-6a5e-4200-8e9f-bc78773e791d-kube-api-access-v89gk\" (UniqueName: \"kubernetes.io/projected/d17d828e-6a5e-4200-8e9f-bc78773e791d-kube-api-access-v89gk\") pod \"d17d828e-6a5e-4200-8e9f-bc78773e791d\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") "
Nov 03 19:48:50.252827 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.252707    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-xks-ca\") pod \"d17d828e-6a5e-4200-8e9f-bc78773e791d\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") "
Nov 03 19:48:50.252874 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.252839    4489 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-openshift-ca\") pod \"d17d828e-6a5e-4200-8e9f-bc78773e791d\" (UID: \"d17d828e-6a5e-4200-8e9f-bc78773e791d\") "
Nov 03 19:48:50.253202 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.253174    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-openshift-ca" pod "d17d828e-6a5e-4200-8e9f-bc78773e791d" (UID: "d17d828e-6a5e-4200-8e9f-bc78773e791d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:50.254963 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.254941    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-xks-ca" pod "d17d828e-6a5e-4200-8e9f-bc78773e791d" (UID: "d17d828e-6a5e-4200-8e9f-bc78773e791d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:50.255026 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.254982    4489 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17d828e-6a5e-4200-8e9f-bc78773e791d-kube-api-access-v89gk" pod "d17d828e-6a5e-4200-8e9f-bc78773e791d" (UID: "d17d828e-6a5e-4200-8e9f-bc78773e791d"). InnerVolumeSpecName "kube-api-access-v89gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:50.354565 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.354487    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-openshift-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:50.354565 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.354515    4489 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89gk\" (UniqueName: \"kubernetes.io/projected/d17d828e-6a5e-4200-8e9f-bc78773e791d-kube-api-access-v89gk\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:50.354565 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:50.354525    4489 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17d828e-6a5e-4200-8e9f-bc78773e791d-maas-api-xks-ca\") on node \"ip-10-0-143-45.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:51.023664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:51.023632    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb" event={"ID":"d17d828e-6a5e-4200-8e9f-bc78773e791d","Type":"ContainerDied","Data":"3d26193a159c97f65860401baac1a285b0efc3588cf32c599c22bd2c988e1917"}
Nov 03 19:48:51.023664 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:51.023653    4489 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-2qfzb"
Nov 03 19:48:51.023850 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:51.023686    4489 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d26193a159c97f65860401baac1a285b0efc3588cf32c599c22bd2c988e1917"
Nov 03 19:48:59.359854 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:48:59.359822    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757fd8969f-t8vmb_dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5/authorino/0.log"
Nov 03 19:49:12.145353 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:12.145285    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q5hdm_2c4f42a8-138c-48c0-8e51-75f7d4a1e302/proxy/0.log"
Nov 03 19:49:13.919374 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:13.919346    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hjx7x_719978d4-84a9-4431-b3f4-e06c34c71ddb/revoke-keys/0.log"
Nov 03 19:49:14.153180 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:14.153132    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-2qfzb_d17d828e-6a5e-4200-8e9f-bc78773e791d/revoke-keys/0.log"
Nov 03 19:49:14.374855 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:14.374777    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-qjlng_8116cdb1-0ed5-477b-91c5-e7a38e6667e8/revoke-keys/0.log"
Nov 03 19:49:16.059574 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:16.059543    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757fd8969f-t8vmb_dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5/authorino/0.log"
Nov 03 19:49:17.085042 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:17.085012    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-tjbps_1deac7e7-8a39-45db-80fd-7ef84c90b418/kuadrant-console-plugin/0.log"
Nov 03 19:49:17.529634 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:49:17.529601    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:49:19.038422 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:19.038389    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-79xhw_f67ee830-9aa5-4ad7-9741-38b9645c6639/discovery/0.log"
Nov 03 19:49:19.422291 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:19.422270    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8f876b79-4bl78_5d350369-34fb-41cd-b1dd-11dfcaabef4f/kube-auth-proxy/0.log"
Nov 03 19:49:20.143090 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:20.143062    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-8vt58_0f19678a-71fc-4cd3-bc44-085860a485cf/istio-proxy/0.log"
Nov 03 19:49:21.649221 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:21.649191    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-v7mn2_6ddb111d-2ea5-43a9-8362-1122f85f28ac/istio-proxy/0.log"
Nov 03 19:49:22.219345 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:22.219313    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-zlt64_f837beb4-df93-41a8-ad75-0381fc6c0995/istio-proxy/0.log"
Nov 03 19:49:24.002085 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:24.002056    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9_cf4216e6-0db2-41a5-8fdd-91320e6681ea/main/0.log"
Nov 03 19:49:24.277073 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:24.276952    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k44p2_0cb6520e-6a2d-43c6-91d8-a19e784b3180/main/0.log"
Nov 03 19:49:24.810365 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:24.810335    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rh6nq_91928076-5652-40ca-ba0e-7cfe3ead9cb6/main/0.log"
Nov 03 19:49:25.303995 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:25.303966    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hjb4p_edeeb111-376a-40b7-85de-3d95677ad5d4/main/0.log"
Nov 03 19:49:25.517602 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:25.517572    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf_a1a211ed-737a-4f7b-bc35-fd1021ac8098/main/0.log"
Nov 03 19:49:25.736191 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:25.736165    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v_e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6/main/0.log"
Nov 03 19:49:26.786156 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:26.786104    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757fd8969f-t8vmb_dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5/authorino/0.log"
Nov 03 19:49:37.948826 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:37.948789    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q5hdm_2c4f42a8-138c-48c0-8e51-75f7d4a1e302/proxy/0.log"
Nov 03 19:49:39.197633 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:39.197603    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hjx7x_719978d4-84a9-4431-b3f4-e06c34c71ddb/revoke-keys/0.log"
Nov 03 19:49:39.431610 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:39.431579    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-2qfzb_d17d828e-6a5e-4200-8e9f-bc78773e791d/revoke-keys/0.log"
Nov 03 19:49:39.663465 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:39.663437    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-qjlng_8116cdb1-0ed5-477b-91c5-e7a38e6667e8/revoke-keys/0.log"
Nov 03 19:49:41.088306 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:41.088278    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757fd8969f-t8vmb_dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5/authorino/0.log"
Nov 03 19:49:42.098372 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:42.098346    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-tjbps_1deac7e7-8a39-45db-80fd-7ef84c90b418/kuadrant-console-plugin/0.log"
Nov 03 19:49:43.738060 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:43.738030    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-79xhw_f67ee830-9aa5-4ad7-9741-38b9645c6639/discovery/0.log"
Nov 03 19:49:44.114499 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:44.114394    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8f876b79-4bl78_5d350369-34fb-41cd-b1dd-11dfcaabef4f/kube-auth-proxy/0.log"
Nov 03 19:49:44.833092 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:44.833047    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-8vt58_0f19678a-71fc-4cd3-bc44-085860a485cf/istio-proxy/0.log"
Nov 03 19:49:46.400130 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:46.400096    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-v7mn2_6ddb111d-2ea5-43a9-8362-1122f85f28ac/istio-proxy/0.log"
Nov 03 19:49:46.934459 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:46.934416    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-zlt64_f837beb4-df93-41a8-ad75-0381fc6c0995/istio-proxy/0.log"
Nov 03 19:49:47.529108 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:49:47.529074    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:49:48.501700 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:48.501657    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-psdk9_cf4216e6-0db2-41a5-8fdd-91320e6681ea/main/0.log"
Nov 03 19:49:48.786336 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:48.786229    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k44p2_0cb6520e-6a2d-43c6-91d8-a19e784b3180/main/0.log"
Nov 03 19:49:49.319632 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:49.319606    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rh6nq_91928076-5652-40ca-ba0e-7cfe3ead9cb6/main/0.log"
Nov 03 19:49:49.814436 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:49.814405    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hjb4p_edeeb111-376a-40b7-85de-3d95677ad5d4/main/0.log"
Nov 03 19:49:50.102637 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:50.102547    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ctzdf_a1a211ed-737a-4f7b-bc35-fd1021ac8098/main/0.log"
Nov 03 19:49:50.329209 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:49:50.329183    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-gh69v_e0f53560-f7a4-4a8b-b908-f1c7a5ce1bd6/main/0.log"
Nov 03 19:50:17.529369 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:50:17.529330    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:50:47.528894 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:50:47.528811    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:51:09.633414 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:09.633386    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xr6hd_793f3faa-8090-4008-a15c-13202c13e875/global-pull-secret-syncer/0.log"
Nov 03 19:51:12.160450 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:12.160418    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b2gtz_43940503-68c4-41e5-9656-4e1fe7837110/konnectivity-agent/0.log"
Nov 03 19:51:14.072440 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:14.072412    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-45.us-west-2.compute.internal_6e273692ac2c0053e598770db34f2c6e/haproxy/0.log"
Nov 03 19:51:17.531554 ip-10-0-143-45 kubenswrapper[4489]: E1103 19:51:17.531501    4489 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n9565" containerName="extractor"
Nov 03 19:51:17.771712 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:17.771677    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757fd8969f-t8vmb_dde2f2d2-a1c6-4b6a-a542-a4e655cd04c5/authorino/0.log"
Nov 03 19:51:18.725069 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.725035    4489 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"]
Nov 03 19:51:18.725498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.725407    4489 cpu_manager.go:414] "containerMap: removing container" podUID="d17d828e-6a5e-4200-8e9f-bc78773e791d" containerName="revoke-keys"
Nov 03 19:51:18.725498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.725418    4489 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17d828e-6a5e-4200-8e9f-bc78773e791d" containerName="revoke-keys"
Nov 03 19:51:18.725498 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.725492    4489 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17d828e-6a5e-4200-8e9f-bc78773e791d" containerName="revoke-keys"
Nov 03 19:51:18.728480 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.728464    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.768352 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.768315    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-sys\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.768483 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.768360    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-proc\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.768537 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.768501    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-podres\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.768593 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.768563    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-lib-modules\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.768692 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.768671    4489 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7s5z\" (UniqueName: \"kubernetes.io/projected/23716844-98a9-4249-8076-92200b791490-kube-api-access-m7s5z\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.782893 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.782838    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjsjt\"/\"kube-root-ca.crt\""
Nov 03 19:51:18.782893 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.782864    4489 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjsjt\"/\"openshift-service-ca.crt\""
Nov 03 19:51:18.834995 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.834959    4489 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tjsjt\"/\"default-dockercfg-89plj\""
Nov 03 19:51:18.836821 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.836802    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"]
Nov 03 19:51:18.869573 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869546    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7s5z\" (UniqueName: \"kubernetes.io/projected/23716844-98a9-4249-8076-92200b791490-kube-api-access-m7s5z\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869670 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869585    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-sys\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869670 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869604    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-proc\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869669    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-podres\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869675    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-sys\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869747 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869727    4489 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-lib-modules\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869747    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-podres\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869754    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-proc\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.869852 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.869827    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23716844-98a9-4249-8076-92200b791490-lib-modules\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.932330 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.932313    4489 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7s5z\" (UniqueName: \"kubernetes.io/projected/23716844-98a9-4249-8076-92200b791490-kube-api-access-m7s5z\") pod \"perf-node-gather-daemonset-r6gd2\" (UID: \"23716844-98a9-4249-8076-92200b791490\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:18.947936 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:18.947916    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-tjbps_1deac7e7-8a39-45db-80fd-7ef84c90b418/kuadrant-console-plugin/0.log"
Nov 03 19:51:19.039700 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:19.039640    4489 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:19.411552 ip-10-0-143-45 kubenswrapper[4489]: W1103 19:51:19.411524    4489 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23716844_98a9_4249_8076_92200b791490.slice/crio-279b919b3039fad4c2e0493fa6fd770d1894ead7b601d9926fdd8a862645b242 WatchSource:0}: Error finding container 279b919b3039fad4c2e0493fa6fd770d1894ead7b601d9926fdd8a862645b242: Status 404 returned error can't find the container with id 279b919b3039fad4c2e0493fa6fd770d1894ead7b601d9926fdd8a862645b242
Nov 03 19:51:19.414590 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:19.414565    4489 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"]
Nov 03 19:51:19.700612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:19.700515    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2" event={"ID":"23716844-98a9-4249-8076-92200b791490","Type":"ContainerStarted","Data":"0b9480d92e2cfacc8abd2250072bf5e2b6167e4dc98100976e2f9a62a299290b"}
Nov 03 19:51:19.700612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:19.700572    4489 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2" event={"ID":"23716844-98a9-4249-8076-92200b791490","Type":"ContainerStarted","Data":"279b919b3039fad4c2e0493fa6fd770d1894ead7b601d9926fdd8a862645b242"}
Nov 03 19:51:19.700612 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:19.700601    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:19.820224 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:19.820174    4489 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2" podStartSLOduration=1.820160647 podStartE2EDuration="1.820160647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:19.82009033 +0000 UTC m=+4408.968784902" watchObservedRunningTime="2026-11-03 19:51:19.820160647 +0000 UTC m=+4408.968855208"
Nov 03 19:51:25.716022 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:25.715996    4489 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-r6gd2"
Nov 03 19:51:28.122788 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:28.122756    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-5svjz_d4e93fd2-3183-488f-b9ba-15cfe67b19a2/monitoring-plugin/0.log"
Nov 03 19:51:31.082748 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:31.082711    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w9jfb_c6efa29b-d526-4d3c-b23c-b21435f0c72c/node-exporter/0.log"
Nov 03 19:51:31.552263 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:31.552231    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w9jfb_c6efa29b-d526-4d3c-b23c-b21435f0c72c/kube-rbac-proxy/0.log"
Nov 03 19:51:32.006995 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:32.006969    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w9jfb_c6efa29b-d526-4d3c-b23c-b21435f0c72c/init-textfile/0.log"
Nov 03 19:51:37.435112 ip-10-0-143-45 kubenswrapper[4489]: I1103 19:51:37.435079    4489 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-r9b4h_91ec170f-ad66-4671-9a63-0be41bbc2f00/prometheus-operator-admission-webhook/0.log"