Nov 03 16:06:55.403805 ip-10-0-137-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 16:06:55.403819 ip-10-0-137-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 16:06:55.403829 ip-10-0-137-122 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 16:06:55.404173 ip-10-0-137-122 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 16:07:05.626062 ip-10-0-137-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 16:07:05.626078 ip-10-0-137-122 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 15fe401bc2b044899688bf3f36df29f1 --
Nov 03 16:09:24.746470 ip-10-0-137-122 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 16:09:25.180215 ip-10-0-137-122 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:09:25.180215 ip-10-0-137-122 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 16:09:25.180215 ip-10-0-137-122 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:09:25.180215 ip-10-0-137-122 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:09:25.183711 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183634    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.183711 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183709    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.183711 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183714    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183720    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183724    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183728    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183732    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183736    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183739    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183743    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183746    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183751    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183762    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183766    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183770    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183774    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183777    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183782    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183786    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183790    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183794    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183797    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.183937 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183801    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183805    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183809    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183813    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183817    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183821    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183825    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183829    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183833    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183838    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183843    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183847    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183852    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183856    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183862    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183874    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183879    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183884    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183888    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.184627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183892    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183896    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183900    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183905    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183909    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183914    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183918    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183923    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183927    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183931    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183935    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183939    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183943    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183947    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183951    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183955    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183959    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183963    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183968    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183972    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.185071 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183977    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183983    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183987    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183991    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183995    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.183999    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184003    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184009    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184013    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184018    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184023    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184028    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184033    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184037    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184042    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184047    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184051    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184055    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184060    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184064    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.185773 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184068    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184073    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184077    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184081    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184086    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184090    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184095    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184099    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184103    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184107    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184111    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184119    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184126    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184130    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184135    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184140    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184144    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184149    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184153    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.186678 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184157    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184162    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184166    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184170    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184175    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.184180    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186435    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186447    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186451    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186456    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186460    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186465    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186469    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186473    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186478    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186482    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186486    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186490    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186495    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186499    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.187175 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186503    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186507    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186511    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186515    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186520    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186525    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186529    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186533    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186539    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186543    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186547    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186551    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186557    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186561    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186566    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186570    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186575    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186580    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186584    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.187674 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186589    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186593    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186598    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186602    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186607    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186611    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186616    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186620    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186625    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186629    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186634    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186638    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186642    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186646    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186651    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186655    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186659    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186664    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186668    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186672    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.188142 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186676    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186680    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186684    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186689    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186693    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186697    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186702    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186707    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186711    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186717    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186721    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186726    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186731    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186735    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186739    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186743    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186747    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186752    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186756    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.188633 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186761    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186765    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186769    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186774    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186803    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186822    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186827    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186831    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186834    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186837    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186841    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186844    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186847    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186851    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186854    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186857    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186860    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186864    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186869    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186873    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.189085 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186876    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186879    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186882    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186893    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186898    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186901    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186903    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186906    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186909    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186912    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186914    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186917    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186919    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.186923    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187472    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187479    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187484    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187488    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187492    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.189616 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187495    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187498    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187500    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187503    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187506    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187509    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187511    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187514    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187517    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187520    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187523    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187525    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187528    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187531    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187534    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187536    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187539    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187541    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187545    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187549    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.190060 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187553    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187555    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187558    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187561    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187563    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187566    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187569    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187572    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187574    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187577    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187581    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187584    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187587    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187589    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187592    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187594    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187597    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187599    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187602    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.190627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187605    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187608    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187610    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187613    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187615    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187617    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187620    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187623    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187625    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187628    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187630    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187633    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187635    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187638    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187641    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187643    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187646    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187648    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187651    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.191070 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187654    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187658    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187661    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187664    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187667    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187670    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187673    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187675    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187678    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187681    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187683    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187686    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187688    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187691    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187694    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187696    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187699    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187701    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187704    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187706    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.191544 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187709    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187711    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187713    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187716    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187718    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187721    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187723    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187726    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187728    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187731    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187733    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187735    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187738    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187741    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187744    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187746    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187749    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187752    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187755    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.192024 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187758    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187761    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187764    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.187766    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188809    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188819    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188826    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188830    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188835    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188838    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188842    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188853    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188856    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188859    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188863    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188867    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188870    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188873    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188876    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188879    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188882    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188885    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188890    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 03 16:09:25.192493 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188893    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188896    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188899    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188902    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188906    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188909    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188913    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188916    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.188919    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189289    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189294    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189298    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189301    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189305    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189308    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189311    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189314    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189317    2582 flags.go:64] FLAG: --enable-server="true"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189320    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189324    2582 flags.go:64] FLAG: --event-burst="100"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189328    2582 flags.go:64] FLAG: --event-qps="50"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189331    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189334    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189337    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 03 16:09:25.193031 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189341    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189344    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189347    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189350    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189353    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189356    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189359    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189362    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189365    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189368    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189370    2582 flags.go:64] FLAG: --feature-gates=""
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189375    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189377    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189381    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189385    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189388    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189391    2582 flags.go:64] FLAG: --help="false"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189394    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189398    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189401    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189404    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189407    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189411    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 16:09:25.193618 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189413    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189416    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189419    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189422    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189425    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189428    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189431    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189433    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189436    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189439    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189442    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189445    2582 flags.go:64] FLAG: --lock-file=""
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189448    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189451    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189454    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189459    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189462    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189465    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189467    2582 flags.go:64] FLAG: --logging-format="text"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189473    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189476    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189479    2582 flags.go:64] FLAG: --manifest-url=""
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189482    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189486    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189490    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 16:09:25.194251 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189493    2582 flags.go:64] FLAG: --max-pods="110"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189496    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189499    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189503    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189506    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189509    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189512    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189515    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189522    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189525    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189528    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189531    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189534    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189537    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189540    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189543    2582 flags.go:64] FLAG: --port="10250"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189546    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189548    2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00053910f14216d9e"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189552    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189555    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189557    2582 flags.go:64] FLAG: --register-node="true"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189560    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189564    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189567    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189570    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189573    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 03 16:09:25.194855 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189578    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189582    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189585    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189587    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189591    2582 flags.go:64] FLAG: --runonce="false"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189593    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189597    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189600    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189602    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189605    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189609    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189612    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189615    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189618    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189621    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189624    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189627    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189631    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189634    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189637    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189642    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189645    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189648    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189652    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189655    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 16:09:25.195492 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189658    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189661    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189664    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189669    2582 flags.go:64] FLAG: --v="2"
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189673    2582 flags.go:64] FLAG: --version="false"
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189677    2582 flags.go:64] FLAG: --vmodule=""
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189681    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.189686    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189812    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189816    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189818    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189821    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189824    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189826    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189829    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189831    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189834    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189836    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189839    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189842    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189844    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.196101 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189847    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189849    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189852    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189855    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189857    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189860    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189863    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189865    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189868    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189870    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189873    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189875    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189878    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189880    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189884    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189887    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189890    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189892    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189896    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.196627 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189899    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189901    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189904    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189906    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189909    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189912    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189914    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189917    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189919    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189922    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189924    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189927    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189930    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189933    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189935    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189937    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189940    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189942    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189945    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189948    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.197447 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189950    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189952    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189955    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189958    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189960    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189963    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189965    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189969    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189971    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189974    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189976    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189980    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189983    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189985    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189988    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189991    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189993    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189996    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.189999    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190001    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.198146 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190004    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190006    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190009    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190011    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190014    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190017    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190019    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190022    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190024    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190027    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190030    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190034    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190038    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190041    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190044    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190046    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190049    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190052    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190054    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.198643 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190058    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190061    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190063    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190066    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190071    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190075    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190079    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190083    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190085    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190088    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190091    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190093    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190096    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190099    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190101    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.199108 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.190127    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190159    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190163    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190166    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190169    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190171    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190174    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190177    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190179    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190182    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190184    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190187    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190189    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190192    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190194    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190209    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190212    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190216    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190218    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.199494 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190221    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190223    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190226    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190230    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190232    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190235    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190237    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190240    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190243    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190245    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190248    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190251    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190254    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190256    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190259    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190261    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190264    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190267    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190270    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190272    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.199943 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190275    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190278    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190280    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190283    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190285    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190288    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190290    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190293    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190295    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190298    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190302    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190305    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190308    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190310    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190313    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190317    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190320    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190324    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190328    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190330    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.200422 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190334    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190337    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190339    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190342    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190344    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190347    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190349    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190352    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190354    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190357    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190361    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190364    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190368    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190370    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190373    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190376    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190378    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190381    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190383    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.200896 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190385    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190388    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190390    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190395    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190398    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190400    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190403    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190405    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190409    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190412    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190414    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190417    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190419    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190422    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190424    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190427    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190430    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190433    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190435    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.201364 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190437    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190440    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190443    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190446    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190449    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190451    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190454    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190456    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190459    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.190462    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.198180    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.198211    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.198230    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.198236    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198291    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198296    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198299    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198302    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198305    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198307    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.201810 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198310    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198312    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198315    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198318    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198320    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198323    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198325    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198328    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198331    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198333    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198336    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198339    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198342    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198345    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198348    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198350    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198353    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198355    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198358    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.202383 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198361    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198363    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198367    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198371    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198374    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198378    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198381    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198383    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198386    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198390    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198394    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198398    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198401    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198404    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198406    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198409    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198412    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198414    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198417    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.202818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198419    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198421    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198424    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198426    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198429    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198432    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198435    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198437    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198440    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198442    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198445    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198447    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198450    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198453    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198456    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198458    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198461    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198464    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198467    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.203375 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198469    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198472    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198475    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198477    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198480    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198483    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198486    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198488    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198491    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198494    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198497    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198499    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198502    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198504    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198507    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198509    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198512    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198515    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198518    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198520    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.203818 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198523    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198525    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198528    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198531    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198533    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198536    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198538    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198541    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198543    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198546    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198549    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198552    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198554    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198557    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198560    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198563    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198565    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198568    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198570    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.204374 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198573    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198576    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198578    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198581    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.198610    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198765    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198772    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198776    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198779    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198782    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198785    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198789    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198792    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198794    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198797    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198800    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.204816 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198803    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198806    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198808    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198811    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198813    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198816    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198819    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198821    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198824    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198827    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198829    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198832    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198834    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198837    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198839    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198842    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198846    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198850    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198853    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.205283 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198856    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198859    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198862    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198865    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198868    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198870    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198873    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198876    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198879    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198882    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198886    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198891    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198894    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198897    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198900    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198903    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198905    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198908    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198910    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.205731 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198913    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198916    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198918    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198921    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198923    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198926    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198929    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198931    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198934    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198937    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198939    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198942    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198944    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198947    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198949    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198952    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198954    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198957    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198959    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.206359 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198962    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198965    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198967    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198970    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198973    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198976    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198978    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198981    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198983    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198986    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198988    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198991    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198994    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198996    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.198999    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199001    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199004    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199006    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199009    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199011    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.207068 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199014    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199017    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199019    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199022    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199025    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199027    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199030    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199032    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199035    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199037    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199040    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199042    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199045    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199048    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199050    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199053    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199056    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.207720 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199059    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.199082    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199113    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199117    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199121    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199124    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199127    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199130    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199133    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199136    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199138    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199141    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199144    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199146    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199149    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199152    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:09:25.208363 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199154    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199157    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199160    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199162    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199165    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199168    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199170    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199173    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199176    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199178    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199181    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199183    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199186    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199188    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199191    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199193    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199211    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199214    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199216    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199219    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:09:25.208744 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199222    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199224    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199226    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199229    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199231    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199234    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199236    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199239    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199242    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199246    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199249    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199251    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199254    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199257    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199259    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199261    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199264    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199266    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199269    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199271    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:09:25.209248 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199277    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199282    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199286    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199289    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199293    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199296    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199299    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199302    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199305    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199308    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199311    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199313    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199316    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199318    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199321    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199323    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199326    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199328    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:09:25.209750 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199331    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199333    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199336    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199338    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199341    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199343    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199346    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199348    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199351    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199353    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199356    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199358    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199361    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199363    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199369    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199371    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199374    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199376    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199379    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199382    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:09:25.210168 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199384    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199387    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199389    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199392    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199395    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199397    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199400    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199402    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199405    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199407    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199409    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199413    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199415    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:25.199418    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.199468    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.201944    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.203067    2582 server.go:1014] "Starting client certificate rotation"
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.203169    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:09:25.210662 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.203228    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:09:25.211077 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.205287    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:09:25.211077 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.209228    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:09:25.218405 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.218390    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 03 16:09:25.222381 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.222368    2582 log.go:25] "Validated CRI v1 image API"
Nov 03 16:09:25.223211 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.223175    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 16:09:25.225863 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.225841    2582 fs.go:135] Filesystem UUIDs: map[74ad188d-f11d-473e-97a8-bdfea535610e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e396b55a-86e3-4eff-ac99-4a9ddd441543:/dev/nvme0n1p4]
Nov 03 16:09:25.225918 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.225864    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 16:09:25.231712 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.231603    2582 manager.go:217] Machine: {Timestamp:2026-11-03 16:09:25.230357142 +0000 UTC m=+0.368634093 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096068 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287d1a8f3850709e49007231a82bad SystemUUID:ec287d1a-8f38-5070-9e49-007231a82bad BootID:15fe401b-c2b0-4489-9688-bf3f36df29f1 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ca:8c:2b:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ca:8c:2b:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:2e:80:73:32:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 16:09:25.231712 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.231710    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 16:09:25.231834 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.231778    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 16:09:25.232969 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.232946    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 16:09:25.233089 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.232972    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-122.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 16:09:25.233836 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.233825    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 16:09:25.233868 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.233838    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 16:09:25.233868 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.233863    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:09:25.233926 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.233877    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:09:25.235133 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.235123    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 16:09:25.239829 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.239817    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 16:09:25.239946 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.239936    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 16:09:25.240749 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.240739    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 16:09:25.240783 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.240752    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 16:09:25.240783 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.240762    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 16:09:25.240783 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.240773    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 03 16:09:25.240783 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.240781    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 16:09:25.242194 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.242169    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:09:25.244937 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.244912    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 16:09:25.251885 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.251861    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:09:25.252725 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252713    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 16:09:25.252779 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252743    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 16:09:25.252916 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252904    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 16:09:25.252964 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252921    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 16:09:25.252964 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252927    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 16:09:25.252964 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252933    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 16:09:25.252964 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252938    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 16:09:25.252964 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.252944    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 16:09:25.253605 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.253596    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 16:09:25.253635 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.253605    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 16:09:25.253635 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.253612    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 16:09:25.253635 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.253618    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 16:09:25.253635 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.253626    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 16:09:25.253635 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.253635    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 16:09:25.254359 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.254349    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 16:09:25.254359 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.254359    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 16:09:25.258017 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.258003    2582 server.go:1263] "Started kubelet"
Nov 03 16:09:25.258103 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.258081    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 16:09:25.258149 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.258100    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 16:09:25.258193 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.258173    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 16:09:25.258761 ip-10-0-137-122 systemd[1]: Started Kubernetes Kubelet.
Nov 03 16:09:25.259668 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.259650    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 16:09:25.260110 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.260096    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 16:09:25.262968 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.262950    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 16:09:25.268504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.268486    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 16:09:25.269189 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.269171    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 16:09:25.269300 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.269272    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-122.us-west-2.compute.internal\" not found"
Nov 03 16:09:25.270013 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.269996    2582 factory.go:55] Registering systemd factory
Nov 03 16:09:25.270013 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270014    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 03 16:09:25.270137 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270029    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 16:09:25.270137 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270032    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 16:09:25.270137 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270053    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 16:09:25.270137 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270109    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 16:09:25.270137 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270117    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 16:09:25.270376 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270240    2582 factory.go:153] Registering CRI-O factory
Nov 03 16:09:25.270376 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270254    2582 factory.go:223] Registration of the crio container factory successfully
Nov 03 16:09:25.270376 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270317    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 16:09:25.270376 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270336    2582 factory.go:103] Registering Raw factory
Nov 03 16:09:25.270376 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.270350    2582 manager.go:1199] Started watching for new ooms in manager
Nov 03 16:09:25.271366 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.271351    2582 manager.go:319] Starting recovery of all containers
Nov 03 16:09:25.282936 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.282910    2582 manager.go:324] Recovery completed
Nov 03 16:09:25.288226 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.288210    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:09:25.290223 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.290188    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:09:25.290295 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.290237    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:09:25.290295 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.290248    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:09:25.290883 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.290870    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 03 16:09:25.290883 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.290881    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 16:09:25.290966 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.290901    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 16:09:25.290966 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.290907    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:09:25.291619 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.291602    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:09:25.291669 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.291628    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:09:25.291669 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.291638    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:09:25.293166 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.293154    2582 policy_none.go:50] "Start"
Nov 03 16:09:25.293166 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.293168    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 16:09:25.293271 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.293177    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 16:09:25.295230 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.295216    2582 policy_none.go:44] "Start"
Nov 03 16:09:25.320968 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.320945    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 16:09:25.323245 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.323231    2582 manager.go:341] "Starting Device Plugin manager"
Nov 03 16:09:25.323981 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.323962    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 16:09:25.323981 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.323981    2582 server.go:87] "Starting device plugin registration server"
Nov 03 16:09:25.324230 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.324218    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 16:09:25.324266 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.324233    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 16:09:25.324330 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.324313    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 16:09:25.324413 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.324400    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 16:09:25.324502 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.324414    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 16:09:25.324838 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.324817    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 16:09:25.324891 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.324864    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-122.us-west-2.compute.internal\" not found"
Nov 03 16:09:25.391797 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.391771    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-122.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 16:09:25.393775 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.391769    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-122.us-west-2.compute.internal.18e494ce45f23b1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-122.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-122.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 16:09:25.257976602 +0000 UTC m=+0.396253554,LastTimestamp:2026-11-03 16:09:25.257976602 +0000 UTC m=+0.396253554,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-122.us-west-2.compute.internal,}"
Nov 03 16:09:25.397770 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.397748    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 16:09:25.399000 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.398985    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 16:09:25.399064 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.399004    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 16:09:25.399064 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.399024    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 16:09:25.399064 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.399060    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 16:09:25.424415 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.424394    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:09:25.425188 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.425170    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:09:25.425285 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.425213    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:09:25.425285 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.425228    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:09:25.425285 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.425262    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.437298 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.437257    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-122.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 16:09:25.458607 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.458480    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 16:09:25.458713 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.458529    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-122.us-west-2.compute.internal.18e494ce47de3cc0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-122.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-122.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-122.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 16:09:25.290220736 +0000 UTC m=+0.428497690,LastTimestamp:2026-11-03 16:09:25.290220736 +0000 UTC m=+0.428497690,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-122.us-west-2.compute.internal,}"
Nov 03 16:09:25.473413 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.473394    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 16:09:25.473497 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.473448    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fwr8r"
Nov 03 16:09:25.499906 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.499886    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"]
Nov 03 16:09:25.499964 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.499936    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:09:25.501100 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.501077    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:09:25.501100 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.501104    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:09:25.501239 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.501114    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:09:25.503274 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503262    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:09:25.503409 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.503448 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503424    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:09:25.503942 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503928    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:09:25.503942 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503936    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:09:25.504043 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503953    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:09:25.504043 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503958    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:09:25.504043 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503963    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:09:25.504043 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.503969    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:09:25.506119 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.506105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.506209 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.506127    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:09:25.506803 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.506776    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:09:25.506803 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.506801    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:09:25.506922 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.506811    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:09:25.534687 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.534670    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-122.us-west-2.compute.internal\" not found" node="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.537211 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.537181    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-122.us-west-2.compute.internal\" not found" node="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.549302 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.549285    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:09:25.549375 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.549334    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:09:25.550246 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.550231    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:09:25.571446 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.571424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.571531 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.571452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9d1cb113f9a0ff39835d34e4e74ebc1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"d9d1cb113f9a0ff39835d34e4e74ebc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.571531 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.571478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.600181 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.600157    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.600181 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:25.600177    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-122.us-west-2.compute.internal\": node \"ip-10-0-137-122.us-west-2.compute.internal\" not found"
Nov 03 16:09:25.669443 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.669427    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.671600 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.671586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.671659 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.671609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.671659 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.671625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9d1cb113f9a0ff39835d34e4e74ebc1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"d9d1cb113f9a0ff39835d34e4e74ebc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.671729 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.671678    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9d1cb113f9a0ff39835d34e4e74ebc1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"d9d1cb113f9a0ff39835d34e4e74ebc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.671729 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.671696    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.671729 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.671708    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.779549 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.779509    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:09:25.780250 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.780239    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.836355 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.836338    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:09:25.837391 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.837379    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:25.840380 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:25.840359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:09:26.203752 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.203720    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 16:09:26.204131 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.203815    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 16:09:26.204131 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.203843    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 16:09:26.204131 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.203896    2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7ccfbe3a02a843d98ab5cd19bd85dc7-f84d5194377c4192.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-122.us-west-2.compute.internal.18e494ce47de3cc0\": read tcp 10.0.137.122:50206->13.184.209.27:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-122.us-west-2.compute.internal.18e494ce47de3cc0  default   4795 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-122.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-122.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-122.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 16:09:25 +0000 UTC,LastTimestamp:2026-11-03 16:09:25.501091741 +0000 UTC m=+0.639368693,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-122.us-west-2.compute.internal,}"
Nov 03 16:09:26.242371 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.242353    2582 apiserver.go:52] "Watching apiserver"
Nov 03 16:09:26.268698 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.268670    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 16:09:26.478078 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.478022    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:04:25 +0000 UTC" deadline="2028-06-25 20:31:32.679927184 +0000 UTC"
Nov 03 16:09:26.478078 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.478047    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14404h22m6.201882929s"
Nov 03 16:09:26.518324 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.518301    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 16:09:26.518591 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.518574    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-sg6b8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7","openshift-cluster-node-tuning-operator/tuned-mlq76","openshift-dns/node-resolver-7nqs4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal","openshift-multus/multus-58crm","openshift-multus/multus-additional-cni-plugins-b7rgl","openshift-multus/network-metrics-daemon-qd8hz","openshift-network-operator/iptables-alerter-dl567","kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal","openshift-image-registry/node-ca-mxwfz","openshift-network-diagnostics/network-check-target-cs98m","openshift-ovn-kubernetes/ovnkube-node-ljp5j","kube-system/global-pull-secret-syncer-h7sv8"]
Nov 03 16:09:26.521115 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.521097    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.522672 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.522650    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:09:26.523455 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.523424    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.525648 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.525633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.527803 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.527786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:26.527906 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.527882    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:26.530019 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.530004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.532204 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.532186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.534528 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.534509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:26.534605 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.534575    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:26.536945 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.536928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.539106 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.539089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.541481 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.541466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.543779 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.543760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.546162 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.546143    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.546250 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.546193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:26.571565 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.571545    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 16:09:26.575829 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.575805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.575951 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.575848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.575951 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.575921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysconfig\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.576073 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.575963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.576073 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.575996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cnibin\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.576073 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.576073 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-device-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-sys-fs\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwhh\" (UniqueName: \"kubernetes.io/projected/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-kube-api-access-dfwhh\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576137    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-netns\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-ovn\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-bin\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576265 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovn-node-metrics-cert\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-var-lib-kubelet\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cnibin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-slash\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-lib-modules\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-daemon-config\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-systemd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576512 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-config\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576812 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-env-overrides\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576812 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-socket-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.576812 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrmz\" (UniqueName: \"kubernetes.io/projected/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kube-api-access-cxrmz\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.576812 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nttb7\" (UniqueName: \"kubernetes.io/projected/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-kube-api-access-nttb7\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.576812 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-node-log\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.576812 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/203b34a5-c159-485a-83fd-34cf7c348b9d-serviceca\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.576812 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-kubelet-config\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-modprobe-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-run\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43f18aa2-9718-4d99-9e5c-0da5f272cd02-host-slash\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-hosts-file\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-tmp-dir\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-kubernetes\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.576980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577003    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cni-binary-copy\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-k8s-cni-cncf-io\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577053    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577127    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 16:09:26.577130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-dbus\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d79060-9c22-412b-b6df-576c7a2dd340-agent-certs\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577240    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577271    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/203b34a5-c159-485a-83fd-34cf7c348b9d-host\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577331    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x72k5\""
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577348    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kkrgk\""
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sfl\" (UniqueName: \"kubernetes.io/projected/203b34a5-c159-485a-83fd-34cf7c348b9d-kube-api-access-57sfl\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577374    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-sys\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-multus\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnzm\" (UniqueName: \"kubernetes.io/projected/43f18aa2-9718-4d99-9e5c-0da5f272cd02-kube-api-access-frnzm\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-netns\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-etc-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qpk4\" (UniqueName: \"kubernetes.io/projected/6f2e70e1-8800-4ab9-9678-34397afb1b9a-kube-api-access-6qpk4\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-tmp\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-kubelet\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.577819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43f18aa2-9718-4d99-9e5c-0da5f272cd02-iptables-alerter-script\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-conf\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-systemd\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-bin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-systemd-units\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-var-lib-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-system-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-hostroot\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.577984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-conf-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-registration-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9g7\" (UniqueName: \"kubernetes.io/projected/64a4a725-bb12-428a-a3d0-62622448f577-kube-api-access-hh9g7\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-host\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-socket-dir-parent\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-multus-certs\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.578626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-tuned\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-etc-kubernetes\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvcbz\" (UniqueName: \"kubernetes.io/projected/c47cd951-b547-4a56-b2c2-1f28b1358ed6-kube-api-access-rvcbz\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578311    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5p8l\" (UniqueName: \"kubernetes.io/projected/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-kube-api-access-l5p8l\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-kubelet\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-log-socket\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d79060-9c22-412b-b6df-576c7a2dd340-konnectivity-ca\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-netd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.579122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.578543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-script-lib\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.579490 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.579195    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:09:26.579490 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.579229    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9gv4t\""
Nov 03 16:09:26.579490 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.579241    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 16:09:26.623280 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623260    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 16:09:26.623374 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623359    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 16:09:26.623414 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623405    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 16:09:26.623465 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623453    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 16:09:26.623514 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623480    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4vrm9\""
Nov 03 16:09:26.623514 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623503    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xhvl5\""
Nov 03 16:09:26.623632 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623537    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 16:09:26.623632 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623592    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxk2h\""
Nov 03 16:09:26.623632 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623606    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 16:09:26.623632 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623626    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 16:09:26.623820 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623638    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4xv66\""
Nov 03 16:09:26.623820 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623625    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:09:26.623820 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623687    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 16:09:26.623820 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.623682    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 16:09:26.624023 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624009    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tk7qx\""
Nov 03 16:09:26.624117 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624100    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xzbs4\""
Nov 03 16:09:26.624182 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624108    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 16:09:26.624182 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624129    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 16:09:26.624182 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624121    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 16:09:26.624343 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624193    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 16:09:26.624343 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624256    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 16:09:26.624343 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624273    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 16:09:26.624343 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624288    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 16:09:26.624343 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.624296    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 16:09:26.625906 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.625891    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 16:09:26.626535 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.626518    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 16:09:26.626613 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.626577    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qnmxw"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-conf\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-systemd\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679615    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-systemd\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-bin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-systemd-units\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-var-lib-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-conf\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-system-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679810    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-systemd-units\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-hostroot\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679853    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-system-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-conf-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679912    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-var-lib-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679952    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-hostroot\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-registration-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.679991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9g7\" (UniqueName: \"kubernetes.io/projected/64a4a725-bb12-428a-a3d0-62622448f577-kube-api-access-hh9g7\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:26.681200 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680007    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-bin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-host\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-socket-dir-parent\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680056    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-registration-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-multus-certs\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680187    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-socket-dir-parent\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-tuned\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680524    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-host\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680569    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-conf-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680780    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680841    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-etc-kubernetes\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcbz\" (UniqueName: \"kubernetes.io/projected/c47cd951-b547-4a56-b2c2-1f28b1358ed6-kube-api-access-rvcbz\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5p8l\" (UniqueName: \"kubernetes.io/projected/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-kube-api-access-l5p8l\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.680978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-kubelet\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-log-socket\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.682093 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d79060-9c22-412b-b6df-576c7a2dd340-konnectivity-ca\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-netd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-script-lib\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysconfig\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cnibin\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-device-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-sys-fs\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwhh\" (UniqueName: \"kubernetes.io/projected/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-kube-api-access-dfwhh\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-netns\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-ovn\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-bin\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683027 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovn-node-metrics-cert\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-var-lib-kubelet\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cnibin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-slash\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-lib-modules\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-daemon-config\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-systemd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-config\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-env-overrides\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-socket-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.681980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrmz\" (UniqueName: \"kubernetes.io/projected/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kube-api-access-cxrmz\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nttb7\" (UniqueName: \"kubernetes.io/projected/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-kube-api-access-nttb7\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-node-log\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/203b34a5-c159-485a-83fd-34cf7c348b9d-serviceca\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-kubelet-config\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-modprobe-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-run\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43f18aa2-9718-4d99-9e5c-0da5f272cd02-host-slash\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-hosts-file\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-tmp-dir\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-kubernetes\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682336    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-multus-certs\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cni-binary-copy\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682415    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-slash\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682490    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-var-lib-kubelet\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682749    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cnibin\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682644    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-kubelet\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682653    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-sys-fs\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-k8s-cni-cncf-io\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-dbus\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.684621 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d79060-9c22-412b-b6df-576c7a2dd340-agent-certs\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/203b34a5-c159-485a-83fd-34cf7c348b9d-host\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683038    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683155    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-modprobe-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683228    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-run\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683237    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d79060-9c22-412b-b6df-576c7a2dd340-konnectivity-ca\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683375    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683477    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-script-lib\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683545    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/203b34a5-c159-485a-83fd-34cf7c348b9d-serviceca\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683564    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-netd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sfl\" (UniqueName: \"kubernetes.io/projected/203b34a5-c159-485a-83fd-34cf7c348b9d-kube-api-access-57sfl\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-sys\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683685    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-multus\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnzm\" (UniqueName: \"kubernetes.io/projected/43f18aa2-9718-4d99-9e5c-0da5f272cd02-kube-api-access-frnzm\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-netns\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683788    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-config\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-etc-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.685504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683813    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpk4\" (UniqueName: \"kubernetes.io/projected/6f2e70e1-8800-4ab9-9678-34397afb1b9a-kube-api-access-6qpk4\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683878    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysconfig\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683878    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-kubernetes\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682703    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-node-log\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.683920    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-etc-kubernetes\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682785    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-ovn\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.682509    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-bin\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684034    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-device-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684044    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-etc-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684090    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-env-overrides\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684052    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/203b34a5-c159-485a-83fd-34cf7c348b9d-host\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684162    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-systemd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43f18aa2-9718-4d99-9e5c-0da5f272cd02-host-slash\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684175    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-lib-modules\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684307    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-log-socket\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684328    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-k8s-cni-cncf-io\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.686308 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684404    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684770    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684913    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-tuned\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.684957    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-kubelet-config\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-tmp\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-kubelet\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43f18aa2-9718-4d99-9e5c-0da5f272cd02-iptables-alerter-script\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685301    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-tmp-dir\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685318    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovn-node-metrics-cert\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685340    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cni-binary-copy\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685376    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-hosts-file\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685398    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-kubelet\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.685496    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.685568    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret podName:2b0366db-bfff-4d9c-af6e-37b79c084739 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:27.185547904 +0000 UTC m=+2.323824879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret") pod "global-pull-secret-syncer-h7sv8" (UID: "2b0366db-bfff-4d9c-af6e-37b79c084739") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-netns\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685652    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.686933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685678    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685814    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-dbus\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.685908    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-multus\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686197    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cnibin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686245    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-sys\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686279    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686284    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-socket-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.686362    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.686413    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:27.186394798 +0000 UTC m=+2.324671758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686457    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-netns\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686497    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43f18aa2-9718-4d99-9e5c-0da5f272cd02-iptables-alerter-script\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686501    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-daemon-config\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.687447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.686845    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.688593 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.688570    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-tmp\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.688775 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.688757    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d79060-9c22-412b-b6df-576c7a2dd340-agent-certs\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.741161 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.741102    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9g7\" (UniqueName: \"kubernetes.io/projected/64a4a725-bb12-428a-a3d0-62622448f577-kube-api-access-hh9g7\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:26.742551 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.742531    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nttb7\" (UniqueName: \"kubernetes.io/projected/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-kube-api-access-nttb7\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.742807 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.742789    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5p8l\" (UniqueName: \"kubernetes.io/projected/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-kube-api-access-l5p8l\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.743961 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.743935    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:09:26.744061 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.743967    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:09:26.744061 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.743981    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zpljr for pod openshift-network-diagnostics/network-check-target-cs98m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:26.744061 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:26.744058    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr podName:a16b1c0a-5c06-4fef-9a5a-664d92a24a25 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:27.244038734 +0000 UTC m=+2.382315699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zpljr" (UniqueName: "kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr") pod "network-check-target-cs98m" (UID: "a16b1c0a-5c06-4fef-9a5a-664d92a24a25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:26.745106 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.745081    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwhh\" (UniqueName: \"kubernetes.io/projected/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-kube-api-access-dfwhh\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.745928 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.745898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrmz\" (UniqueName: \"kubernetes.io/projected/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kube-api-access-cxrmz\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.746033 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.745963    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpk4\" (UniqueName: \"kubernetes.io/projected/6f2e70e1-8800-4ab9-9678-34397afb1b9a-kube-api-access-6qpk4\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:26.746126 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.746104    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcbz\" (UniqueName: \"kubernetes.io/projected/c47cd951-b547-4a56-b2c2-1f28b1358ed6-kube-api-access-rvcbz\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.746187 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.746122    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sfl\" (UniqueName: \"kubernetes.io/projected/203b34a5-c159-485a-83fd-34cf7c348b9d-kube-api-access-57sfl\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.746187 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.746144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnzm\" (UniqueName: \"kubernetes.io/projected/43f18aa2-9718-4d99-9e5c-0da5f272cd02-kube-api-access-frnzm\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.857460 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.857437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:09:26.870676 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.870657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:09:26.881602 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.881584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:09:26.899405 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.899386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58crm"
Nov 03 16:09:26.912601 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.912582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:26.947445 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.947421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:09:26.951250 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.951232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:09:26.966909 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.966891    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:09:26.992835 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:26.992788    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:09:27.021219 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.021178    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff34aa0_6133_41ff_b88a_f5360d6d2f90.slice/crio-de8eec11ad8ce3db1e65028d91b707b2571f2a0ea697158f4d0eccafa5f2dc34 WatchSource:0}: Error finding container de8eec11ad8ce3db1e65028d91b707b2571f2a0ea697158f4d0eccafa5f2dc34: Status 404 returned error can't find the container with id de8eec11ad8ce3db1e65028d91b707b2571f2a0ea697158f4d0eccafa5f2dc34
Nov 03 16:09:27.021667 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.021643    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d1cb113f9a0ff39835d34e4e74ebc1.slice/crio-c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510 WatchSource:0}: Error finding container c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510: Status 404 returned error can't find the container with id c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510
Nov 03 16:09:27.023123 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.023103    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2e70e1_8800_4ab9_9678_34397afb1b9a.slice/crio-dc9350bb5c534b03b6eb51184331bf12244e699ad8f4cf55b5f0bbd32d90cef2 WatchSource:0}: Error finding container dc9350bb5c534b03b6eb51184331bf12244e699ad8f4cf55b5f0bbd32d90cef2: Status 404 returned error can't find the container with id dc9350bb5c534b03b6eb51184331bf12244e699ad8f4cf55b5f0bbd32d90cef2
Nov 03 16:09:27.023479 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.023449    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fe4bd879ae26257e3dc2e6b3fda856.slice/crio-99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954 WatchSource:0}: Error finding container 99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954: Status 404 returned error can't find the container with id 99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954
Nov 03 16:09:27.027461 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.027445    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:09:27.188451 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.188422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:27.188559 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.188477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:27.188559 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.188547    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:27.188681 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.188609    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:28.188590983 +0000 UTC m=+3.326867944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:27.188681 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.188607    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:27.188681 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.188663    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret podName:2b0366db-bfff-4d9c-af6e-37b79c084739 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:28.188646331 +0000 UTC m=+3.326923287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret") pod "global-pull-secret-syncer-h7sv8" (UID: "2b0366db-bfff-4d9c-af6e-37b79c084739") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:27.289525 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.289477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:27.290055 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.289608    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:09:27.290055 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.289623    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:09:27.290055 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.289633    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zpljr for pod openshift-network-diagnostics/network-check-target-cs98m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:27.290055 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:27.289674    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr podName:a16b1c0a-5c06-4fef-9a5a-664d92a24a25 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:28.289659777 +0000 UTC m=+3.427936734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpljr" (UniqueName: "kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr") pod "network-check-target-cs98m" (UID: "a16b1c0a-5c06-4fef-9a5a-664d92a24a25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:27.404794 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.404751    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7nqs4" event={"ID":"7ff34aa0-6133-41ff-b88a-f5360d6d2f90","Type":"ContainerStarted","Data":"de8eec11ad8ce3db1e65028d91b707b2571f2a0ea697158f4d0eccafa5f2dc34"}
Nov 03 16:09:27.405778 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.405755    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"dc9350bb5c534b03b6eb51184331bf12244e699ad8f4cf55b5f0bbd32d90cef2"}
Nov 03 16:09:27.406687 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.406666    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"d9d1cb113f9a0ff39835d34e4e74ebc1","Type":"ContainerStarted","Data":"c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510"}
Nov 03 16:09:27.407637 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.407603    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"f1fe4bd879ae26257e3dc2e6b3fda856","Type":"ContainerStarted","Data":"99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954"}
Nov 03 16:09:27.550131 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.550086    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:09:27.630154 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.630111    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:04:26 +0000 UTC" deadline="2028-08-07 02:28:37.828189058 +0000 UTC"
Nov 03 16:09:27.630261 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.630161    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15418h19m10.19803042s"
Nov 03 16:09:27.762989 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:27.762968    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:09:27.790415 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.790392    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203b34a5_c159_485a_83fd_34cf7c348b9d.slice/crio-61f1def4221d45f9b717029ef082e79ec61eb29161d715aad69795a18a3f4b70 WatchSource:0}: Error finding container 61f1def4221d45f9b717029ef082e79ec61eb29161d715aad69795a18a3f4b70: Status 404 returned error can't find the container with id 61f1def4221d45f9b717029ef082e79ec61eb29161d715aad69795a18a3f4b70
Nov 03 16:09:27.811236 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.811199    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d79060_9c22_412b_b6df_576c7a2dd340.slice/crio-508e37556cb72662a36c5f780d4798bd85b0d31d83c83e7fd23212f11a3e0914 WatchSource:0}: Error finding container 508e37556cb72662a36c5f780d4798bd85b0d31d83c83e7fd23212f11a3e0914: Status 404 returned error can't find the container with id 508e37556cb72662a36c5f780d4798bd85b0d31d83c83e7fd23212f11a3e0914
Nov 03 16:09:27.814582 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.814562    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47cd951_b547_4a56_b2c2_1f28b1358ed6.slice/crio-864b592a1151b62cfee3a739c8a2383ff72aa8031ed2643e8c9501dbb6727a89 WatchSource:0}: Error finding container 864b592a1151b62cfee3a739c8a2383ff72aa8031ed2643e8c9501dbb6727a89: Status 404 returned error can't find the container with id 864b592a1151b62cfee3a739c8a2383ff72aa8031ed2643e8c9501dbb6727a89
Nov 03 16:09:27.815291 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.815251    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfea5a0_fec4_4ddc_a409_57656d6bfb7f.slice/crio-a11b624f51216074852b68e061ed288d3ee79d5564a9ef1cf3ef9c03713287a9 WatchSource:0}: Error finding container a11b624f51216074852b68e061ed288d3ee79d5564a9ef1cf3ef9c03713287a9: Status 404 returned error can't find the container with id a11b624f51216074852b68e061ed288d3ee79d5564a9ef1cf3ef9c03713287a9
Nov 03 16:09:27.837021 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.837001    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb567c0b7_83c8_46f2_8e94_6ab1ccbffb07.slice/crio-38dc55c08c6f3ff3db30be0f0dfedd7c35442208df2b1fe53014d9fef83c5567 WatchSource:0}: Error finding container 38dc55c08c6f3ff3db30be0f0dfedd7c35442208df2b1fe53014d9fef83c5567: Status 404 returned error can't find the container with id 38dc55c08c6f3ff3db30be0f0dfedd7c35442208df2b1fe53014d9fef83c5567
Nov 03 16:09:27.913318 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.913295    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90ec2f4_8ef6_40ec_b47d_c109711dc0ff.slice/crio-f1ef976c87d917640602d55433763af947237fb0c2c8f4f834f01a101a3858b1 WatchSource:0}: Error finding container f1ef976c87d917640602d55433763af947237fb0c2c8f4f834f01a101a3858b1: Status 404 returned error can't find the container with id f1ef976c87d917640602d55433763af947237fb0c2c8f4f834f01a101a3858b1
Nov 03 16:09:27.922858 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:09:27.922836    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f18aa2_9718_4d99_9e5c_0da5f272cd02.slice/crio-0733f0ad5e5ae264f06ec8aa03f457f50b0123458d2580d34fd80209d4d6a2ae WatchSource:0}: Error finding container 0733f0ad5e5ae264f06ec8aa03f457f50b0123458d2580d34fd80209d4d6a2ae: Status 404 returned error can't find the container with id 0733f0ad5e5ae264f06ec8aa03f457f50b0123458d2580d34fd80209d4d6a2ae
Nov 03 16:09:28.195310 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.195234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:28.195310 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.195272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:28.195481 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.195374    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:28.195481 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.195383    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:28.195481 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.195430    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret podName:2b0366db-bfff-4d9c-af6e-37b79c084739 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:30.195413997 +0000 UTC m=+5.333690942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret") pod "global-pull-secret-syncer-h7sv8" (UID: "2b0366db-bfff-4d9c-af6e-37b79c084739") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:28.195481 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.195448    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:30.195438861 +0000 UTC m=+5.333715802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:28.295727 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.295693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:28.296234 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.295816    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:09:28.296234 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.295832    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:09:28.296234 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.295841    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zpljr for pod openshift-network-diagnostics/network-check-target-cs98m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:28.296234 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.295891    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr podName:a16b1c0a-5c06-4fef-9a5a-664d92a24a25 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:30.295874788 +0000 UTC m=+5.434151731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpljr" (UniqueName: "kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr") pod "network-check-target-cs98m" (UID: "a16b1c0a-5c06-4fef-9a5a-664d92a24a25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:28.399594 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.399556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:28.399740 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.399688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:28.399740 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.399711    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:28.399740 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.399701    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:28.399864 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.399754    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:28.399901 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:28.399871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:28.411762 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.411733    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dl567" event={"ID":"43f18aa2-9718-4d99-9e5c-0da5f272cd02","Type":"ContainerStarted","Data":"0733f0ad5e5ae264f06ec8aa03f457f50b0123458d2580d34fd80209d4d6a2ae"}
Nov 03 16:09:28.413036 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.413006    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerStarted","Data":"f1ef976c87d917640602d55433763af947237fb0c2c8f4f834f01a101a3858b1"}
Nov 03 16:09:28.414234 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.414186    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7" event={"ID":"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f","Type":"ContainerStarted","Data":"a11b624f51216074852b68e061ed288d3ee79d5564a9ef1cf3ef9c03713287a9"}
Nov 03 16:09:28.415428 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.415391    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mxwfz" event={"ID":"203b34a5-c159-485a-83fd-34cf7c348b9d","Type":"ContainerStarted","Data":"61f1def4221d45f9b717029ef082e79ec61eb29161d715aad69795a18a3f4b70"}
Nov 03 16:09:28.416560 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.416542    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlq76" event={"ID":"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07","Type":"ContainerStarted","Data":"38dc55c08c6f3ff3db30be0f0dfedd7c35442208df2b1fe53014d9fef83c5567"}
Nov 03 16:09:28.417595 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.417574    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58crm" event={"ID":"c47cd951-b547-4a56-b2c2-1f28b1358ed6","Type":"ContainerStarted","Data":"864b592a1151b62cfee3a739c8a2383ff72aa8031ed2643e8c9501dbb6727a89"}
Nov 03 16:09:28.418731 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.418706    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sg6b8" event={"ID":"d5d79060-9c22-412b-b6df-576c7a2dd340","Type":"ContainerStarted","Data":"508e37556cb72662a36c5f780d4798bd85b0d31d83c83e7fd23212f11a3e0914"}
Nov 03 16:09:28.630630 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.630548    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:04:26 +0000 UTC" deadline="2028-04-04 07:45:01.05484526 +0000 UTC"
Nov 03 16:09:28.630630 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:28.630578    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12423h35m32.4242704s"
Nov 03 16:09:30.210367 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.210093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:30.210367 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.210359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:30.210748 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.210281    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:30.210748 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.210493    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:34.210466649 +0000 UTC m=+9.348743590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:30.210748 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.210513    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:30.210748 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.210571    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret podName:2b0366db-bfff-4d9c-af6e-37b79c084739 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:34.210554841 +0000 UTC m=+9.348831795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret") pod "global-pull-secret-syncer-h7sv8" (UID: "2b0366db-bfff-4d9c-af6e-37b79c084739") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:30.311212 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.311173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:30.311384 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.311367    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:09:30.311463 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.311391    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:09:30.311463 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.311404    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zpljr for pod openshift-network-diagnostics/network-check-target-cs98m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:30.311548 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.311463    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr podName:a16b1c0a-5c06-4fef-9a5a-664d92a24a25 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:34.311444623 +0000 UTC m=+9.449721575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpljr" (UniqueName: "kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr") pod "network-check-target-cs98m" (UID: "a16b1c0a-5c06-4fef-9a5a-664d92a24a25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:30.399950 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.399928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:30.400082 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.399967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:30.400082 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.400048    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:30.400179 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.400123    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:30.400179 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.400164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:30.400266 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:30.400237    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:30.425983 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.425954    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"d9d1cb113f9a0ff39835d34e4e74ebc1","Type":"ContainerStarted","Data":"836e1ea7893f7572df8280a0a7c58c039a0a431a8f33e326e161a446d6dc28fb"}
Nov 03 16:09:30.536755 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:30.536678    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal" podStartSLOduration=4.536665566 podStartE2EDuration="4.536665566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:30.536440099 +0000 UTC m=+5.674717062" watchObservedRunningTime="2026-11-03 16:09:30.536665566 +0000 UTC m=+5.674942528"
Nov 03 16:09:32.399438 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:32.399403    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:32.399908 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:32.399560    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:32.399908 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:32.399640    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:32.399908 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:32.399721    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:32.399908 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:32.399770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:32.399908 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:32.399837    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:33.433575 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:33.433537    2582 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="b92dc18c7fcd25022d0daecc74de453079167d32c321b0f60188a135cc323021" exitCode=0
Nov 03 16:09:33.434055 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:33.433828    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerDied","Data":"b92dc18c7fcd25022d0daecc74de453079167d32c321b0f60188a135cc323021"}
Nov 03 16:09:33.436286 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:33.436254    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mxwfz" event={"ID":"203b34a5-c159-485a-83fd-34cf7c348b9d","Type":"ContainerStarted","Data":"bdf6af6100ee086728a383f048986bd565b0d74526a68ce4f5640aa65c0d9781"}
Nov 03 16:09:33.438396 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:33.438372    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sg6b8" event={"ID":"d5d79060-9c22-412b-b6df-576c7a2dd340","Type":"ContainerStarted","Data":"24acbe8256a4a4675be93cd4e076cd090e0fe08017e64cc3a2c45517389ea071"}
Nov 03 16:09:33.660062 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:33.660020    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mxwfz" podStartSLOduration=3.938965311 podStartE2EDuration="8.660006086s" totalImagesPullingTime="4.721040775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:33.659816255 +0000 UTC m=+8.798093215" watchObservedRunningTime="2026-11-03 16:09:33.660006086 +0000 UTC m=+8.798283045"
Nov 03 16:09:34.242055 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.241976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:34.242055 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.242027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:34.242261 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.242152    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:34.242261 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.242164    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:34.242261 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.242221    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret podName:2b0366db-bfff-4d9c-af6e-37b79c084739 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:42.242208244 +0000 UTC m=+17.380485195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret") pod "global-pull-secret-syncer-h7sv8" (UID: "2b0366db-bfff-4d9c-af6e-37b79c084739") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:34.242261 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.242236    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:42.242228349 +0000 UTC m=+17.380505289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:34.343332 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.343302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:34.343484 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.343462    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:09:34.343548 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.343491    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:09:34.343548 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.343501    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zpljr for pod openshift-network-diagnostics/network-check-target-cs98m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:34.343616 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.343548    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr podName:a16b1c0a-5c06-4fef-9a5a-664d92a24a25 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:42.343532239 +0000 UTC m=+17.481809178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpljr" (UniqueName: "kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr") pod "network-check-target-cs98m" (UID: "a16b1c0a-5c06-4fef-9a5a-664d92a24a25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:34.399703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.399673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:34.399703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.399690    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:34.399899 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.399723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:34.399899 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.399803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:34.399899 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.399879    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:34.400071 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:34.399971    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:34.441403 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.441375    2582 generic.go:356] "Generic (PLEG): container finished" podID="f1fe4bd879ae26257e3dc2e6b3fda856" containerID="8cb78d9b249a389080e24ca0d0ef5eee947f52af3f29cdaa55f0c58e1ad6d38b" exitCode=0
Nov 03 16:09:34.441739 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.441449    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"f1fe4bd879ae26257e3dc2e6b3fda856","Type":"ContainerDied","Data":"8cb78d9b249a389080e24ca0d0ef5eee947f52af3f29cdaa55f0c58e1ad6d38b"}
Nov 03 16:09:34.551029 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:34.550853    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sg6b8" podStartSLOduration=5.168759241 podStartE2EDuration="9.550839174s" totalImagesPullingTime="4.382079933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:33.773783214 +0000 UTC m=+8.912060169" watchObservedRunningTime="2026-11-03 16:09:34.550839174 +0000 UTC m=+9.689116134"
Nov 03 16:09:35.445898 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:35.445863    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"f1fe4bd879ae26257e3dc2e6b3fda856","Type":"ContainerStarted","Data":"cb4d432a41f4fd82f963595079f4a6526a32a6b4ae37cb4f957bf334f9bcbd7f"}
Nov 03 16:09:35.559114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:35.559054    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal" podStartSLOduration=10.559037659 podStartE2EDuration="10.559037659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:35.55844835 +0000 UTC m=+10.696725312" watchObservedRunningTime="2026-11-03 16:09:35.559037659 +0000 UTC m=+10.697314622"
Nov 03 16:09:36.400058 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:36.400016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:36.400058 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:36.400042    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:36.400324 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:36.400042    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:36.400324 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:36.400148    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:36.400324 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:36.400234    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:36.400324 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:36.400265    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:36.448765 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:36.448737    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7" event={"ID":"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f","Type":"ContainerStarted","Data":"9d63901fc633242e0ab30c347753fae24c7f9a2a5c33a7c5c207cbca463ddfce"}
Nov 03 16:09:38.098487 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.098454    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:38.099073 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.099036    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:38.399941 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.399911    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:38.400102 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.399954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:38.400102 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:38.400025    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:38.400179 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:38.400101    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:38.400179 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.400152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:38.400352 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:38.400325    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:38.454356 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.454324    2582 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="55d8c231f4beb8128c373aa93699cee773a930b362ee63b27c4c556065cefde0" exitCode=0
Nov 03 16:09:38.454495 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.454411    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerDied","Data":"55d8c231f4beb8128c373aa93699cee773a930b362ee63b27c4c556065cefde0"}
Nov 03 16:09:38.455858 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.455832    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58crm" event={"ID":"c47cd951-b547-4a56-b2c2-1f28b1358ed6","Type":"ContainerStarted","Data":"925a62b79cbbbc9467e70e0b56acfc89a1d7507991087a8598e1b21a90f9efaf"}
Nov 03 16:09:38.691951 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:38.691640    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-58crm" podStartSLOduration=3.388504134 podStartE2EDuration="13.691622184s" totalImagesPullingTime="10.30311805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:38.691110304 +0000 UTC m=+13.829387266" watchObservedRunningTime="2026-11-03 16:09:38.691622184 +0000 UTC m=+13.829899147"
Nov 03 16:09:39.702473 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:39.702446    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:39.702793 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:39.702577    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:09:39.703551 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:39.703375    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:09:39.743982 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:39.743962    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 16:09:40.335987 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.335909    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:09:39.743978133Z","UUID":"12ff3ffd-f638-426f-9fcc-ae2eef11c4ae","Handler":null,"Name":"","Endpoint":""}
Nov 03 16:09:40.337099 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.337085    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 16:09:40.337176 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.337108    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 16:09:40.399998 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.399973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:40.400085 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.400032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:40.400085 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.400038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:40.400174 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:40.400116    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:40.400174 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:40.400153    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:40.400265 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:40.400192    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:40.460831 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.460806    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7nqs4" event={"ID":"7ff34aa0-6133-41ff-b88a-f5360d6d2f90","Type":"ContainerStarted","Data":"2896a578f4a91752decd645a53745ba18b30f608401e80ad38c6489cb11fca85"}
Nov 03 16:09:40.462963 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.462944    2582 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="c3a1c8f8f00a9e4cff6781e275bdabf32a056c8a44a1b70024be3c038e15bdf4" exitCode=0
Nov 03 16:09:40.463041 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.463007    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerDied","Data":"c3a1c8f8f00a9e4cff6781e275bdabf32a056c8a44a1b70024be3c038e15bdf4"}
Nov 03 16:09:40.464950 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.464920    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7" event={"ID":"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f","Type":"ContainerStarted","Data":"e10a27afaed6d6207dcd3b6507dea6307710b4c0c38595bdf2cd883d6f6a4a19"}
Nov 03 16:09:40.572857 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:40.572802    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7nqs4" podStartSLOduration=1.985089544 podStartE2EDuration="14.57279161s" totalImagesPullingTime="12.587702066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:40.572603689 +0000 UTC m=+15.710880653" watchObservedRunningTime="2026-11-03 16:09:40.57279161 +0000 UTC m=+15.711068570"
Nov 03 16:09:41.468073 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:41.468037    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dl567" event={"ID":"43f18aa2-9718-4d99-9e5c-0da5f272cd02","Type":"ContainerStarted","Data":"ea3d71d9949d0e1a600899033299eb041e5ec7eb92c40f3134ced335be86a700"}
Nov 03 16:09:41.579521 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:41.579481    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dl567" podStartSLOduration=3.8861931 podStartE2EDuration="15.579460874s" totalImagesPullingTime="11.693267774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:41.578807994 +0000 UTC m=+16.717084956" watchObservedRunningTime="2026-11-03 16:09:41.579460874 +0000 UTC m=+16.717737836"
Nov 03 16:09:42.308835 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:42.308800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:42.308835 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:42.308842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:42.309079 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.308975    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:42.309079 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.308986    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:42.309079 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.309025    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret podName:2b0366db-bfff-4d9c-af6e-37b79c084739 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:58.309011524 +0000 UTC m=+33.447288465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret") pod "global-pull-secret-syncer-h7sv8" (UID: "2b0366db-bfff-4d9c-af6e-37b79c084739") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:42.309079 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.309044    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:58.309034595 +0000 UTC m=+33.447311536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:42.399304 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:42.399278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:42.399424 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:42.399347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:42.399457 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:42.399436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:42.399457 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.399432    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:42.399547 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.399521    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:42.399583 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.399566    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:42.409479 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:42.409457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:42.409628 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.409612    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:09:42.409689 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.409637    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:09:42.409689 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.409649    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zpljr for pod openshift-network-diagnostics/network-check-target-cs98m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:42.409778 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:42.409704    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr podName:a16b1c0a-5c06-4fef-9a5a-664d92a24a25 nodeName:}" failed. No retries permitted until 2026-11-03 16:09:58.409686198 +0000 UTC m=+33.547963154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpljr" (UniqueName: "kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr") pod "network-check-target-cs98m" (UID: "a16b1c0a-5c06-4fef-9a5a-664d92a24a25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:43.473957 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:43.473922    2582 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="b6af3c39d249cefab412729deca5c58340cb4ed54628479f649a79bc754e9f84" exitCode=0
Nov 03 16:09:43.474442 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:43.473972    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerDied","Data":"b6af3c39d249cefab412729deca5c58340cb4ed54628479f649a79bc754e9f84"}
Nov 03 16:09:44.399673 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:44.399586    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:44.399789 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:44.399675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:44.399789 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:44.399701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:44.399789 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:44.399678    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:44.399789 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:44.399760    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:44.399915 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:44.399827    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:44.478234 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:44.478194    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7" event={"ID":"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f","Type":"ContainerStarted","Data":"92efbaeb531e199373c18808c27a17ae7f17ce28d8f24fd45573cfccb55c87a3"}
Nov 03 16:09:44.591823 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:44.591780    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7" podStartSLOduration=3.714522772 podStartE2EDuration="19.591761391s" totalImagesPullingTime="15.877238619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:44.591746802 +0000 UTC m=+19.730023764" watchObservedRunningTime="2026-11-03 16:09:44.591761391 +0000 UTC m=+19.730038353"
Nov 03 16:09:46.400081 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:46.400050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:46.400528 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:46.400135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:46.400528 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:46.400273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:46.400528 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:46.400283    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:46.400528 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:46.400386    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:46.400528 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:46.400469    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:47.485209 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:47.485175    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlq76" event={"ID":"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07","Type":"ContainerStarted","Data":"485043b8438889e59760a88b519c956864b0a30e49a0a411bbfc8806b0365be0"}
Nov 03 16:09:47.597184 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:47.597145    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mlq76" podStartSLOduration=3.5561277479999998 podStartE2EDuration="22.597133334s" totalImagesPullingTime="19.041005586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:47.597018313 +0000 UTC m=+22.735295279" watchObservedRunningTime="2026-11-03 16:09:47.597133334 +0000 UTC m=+22.735410294"
Nov 03 16:09:48.399819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:48.399784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:48.399819 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:48.399815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:48.400021 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:48.399783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:48.400021 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:48.399898    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:48.400021 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:48.399964    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:48.400169 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:48.400027    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:49.490603 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:49.490574    2582 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="fe6f63e6ed661f5fe16dda6d175d4045ba02a3e993544db1bf746d25fed72c81" exitCode=0
Nov 03 16:09:49.490958 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:49.490624    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerDied","Data":"fe6f63e6ed661f5fe16dda6d175d4045ba02a3e993544db1bf746d25fed72c81"}
Nov 03 16:09:50.399230 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:50.399185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:50.399230 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:50.399217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:50.399442 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:50.399185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:50.399442 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:50.399303    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:50.399442 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:50.399381    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:50.399564 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:50.399474    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:50.494975 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:50.494949    2582 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="959e8721a5f76ac90915045eada8912ce0c779512dc71c79855defb7f8c20cdf" exitCode=0
Nov 03 16:09:50.495289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:50.494988    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerDied","Data":"959e8721a5f76ac90915045eada8912ce0c779512dc71c79855defb7f8c20cdf"}
Nov 03 16:09:51.499606 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:51.499574    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" event={"ID":"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff","Type":"ContainerStarted","Data":"3c190686ef4bc982d2a6bf025cf3f1178d49b7c72e2fbc07b6ff0aa420b9b24c"}
Nov 03 16:09:51.710215 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:51.710149    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b7rgl" podStartSLOduration=6.843687621 podStartE2EDuration="25.71013593s" totalImagesPullingTime="18.866448309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:09:51.66278875 +0000 UTC m=+26.801065710" watchObservedRunningTime="2026-11-03 16:09:51.71013593 +0000 UTC m=+26.848412890"
Nov 03 16:09:52.399547 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:52.399519    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:52.399710 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:52.399599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:52.399710 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:52.399678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:52.399710 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:52.399700    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:52.399807 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:52.399735    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:52.399838 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:52.399820    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:54.399568 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:54.399530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:54.399568 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:54.399563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:54.400018 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:54.399643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:54.400018 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:54.399651    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:54.400018 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:54.399726    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:54.400018 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:54.399796    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:56.399571 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:56.399526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:56.400044 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:56.399591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:56.400044 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:56.399610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:56.400044 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:56.399724    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:56.400044 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:56.399805    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:56.400044 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:56.399881    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:58.324805 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:58.324770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:58.325183 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:58.324846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:58.325183 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.324922    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:58.325183 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.324922    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:58.325183 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.324984    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.324966583 +0000 UTC m=+65.463243536 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 16:09:58.325183 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.324998    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret podName:2b0366db-bfff-4d9c-af6e-37b79c084739 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.324992204 +0000 UTC m=+65.463269142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret") pod "global-pull-secret-syncer-h7sv8" (UID: "2b0366db-bfff-4d9c-af6e-37b79c084739") : object "kube-system"/"original-pull-secret" not registered
Nov 03 16:09:58.399561 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:58.399538    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:09:58.399625 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:58.399561    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:09:58.399659 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:58.399539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:58.399696 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.399639    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:09:58.399742 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.399727    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:09:58.399847 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.399823    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:09:58.425083 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:09:58.425058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:09:58.425217 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.425166    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 16:09:58.425217 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.425181    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 16:09:58.425217 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.425190    2582 projected.go:196] Error preparing data for projected volume kube-api-access-zpljr for pod openshift-network-diagnostics/network-check-target-cs98m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:09:58.425326 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:09:58.425243    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr podName:a16b1c0a-5c06-4fef-9a5a-664d92a24a25 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.425231297 +0000 UTC m=+65.563508236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpljr" (UniqueName: "kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr") pod "network-check-target-cs98m" (UID: "a16b1c0a-5c06-4fef-9a5a-664d92a24a25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 16:10:00.399767 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:00.399728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:00.400282 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:00.399801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:00.400282 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:00.399828    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:00.400282 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:00.399945    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:00.400282 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:00.399974    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:00.400282 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:00.400043    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:02.399369 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:02.399334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:02.399369 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:02.399370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:02.399820 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:02.399447    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:02.399820 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:02.399456    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:02.399820 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:02.399535    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:02.399820 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:02.399606    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:04.399515 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:04.399482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:04.399515 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:04.399518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:04.400084 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:04.399598    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:04.400084 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:04.399604    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:04.400084 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:04.399664    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:04.400084 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:04.399729    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:06.400066 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:06.400033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:06.400066 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:06.400067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:06.400559 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:06.400162    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:06.400559 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:06.400236    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:06.400559 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:06.400298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:06.400559 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:06.400399    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:08.399504 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:08.399468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:08.399925 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:08.399511    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:08.399925 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:08.399592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:08.399925 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:08.399599    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:08.399925 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:08.399693    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:08.399925 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:08.399774    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:09.537607 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:09.537494    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"5c15fd33b11b9db720900480aa09dd9da1e14403a7e6d4e252b15902cd956f53"}
Nov 03 16:10:10.400046 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.400005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:10.400294 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.400072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:10.400294 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.400162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:10.400294 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:10.400167    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:10.400294 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:10.400242    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:10.400463 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:10.400333    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:10.542468 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.542294    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"526f176f962225e29ba6537d481212cd028cc3fd25d6242d592585ddd10bacfe"}
Nov 03 16:10:10.542917 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.542502    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"5188e86ca9af668e8372f3fb6994b5e741b0580597d1dffb94b6a80d9f69593b"}
Nov 03 16:10:10.542917 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.542531    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"a8fc8d2420c8e87b790625f6ecbf4ea99e5dc9233551b16fb294afcc80bc546a"}
Nov 03 16:10:10.542917 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.542558    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"9351c7d46d8fa603de7ac4cdd777153e7f29966ffa72e76b56f4b3f406f7ab94"}
Nov 03 16:10:10.542917 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:10.542584    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"08466f27bef381884ae8f8258cfac5a17c44efe117b92c827e899f6c6660d54e"}
Nov 03 16:10:12.399861 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:12.399827    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:12.399861 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:12.399856    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:12.400325 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:12.399933    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:12.400325 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:12.399986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:12.400325 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:12.400050    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:12.400325 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:12.400105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:12.549326 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:12.549293    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"ea4bdd6218e2a0aac2a18ee1b3885d8b04f8b7423fbcd68b8892e1915beadf72"}
Nov 03 16:10:14.399962 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:14.399925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:14.399962 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:14.399950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:14.400475 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:14.399932    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:14.400475 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:14.400033    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:14.400475 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:14.400130    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:14.400475 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:14.400232    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:14.556579 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:14.556538    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" event={"ID":"6f2e70e1-8800-4ab9-9678-34397afb1b9a","Type":"ContainerStarted","Data":"eda74a3a353d220b8a55601e4cb370db4f598e8aeb260a4f76c82211b2f3f310"}
Nov 03 16:10:14.556813 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:14.556796    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:14.572060 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:14.572036    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:14.819987 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:14.819891    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j" podStartSLOduration=6.403317221 podStartE2EDuration="48.819878279s" totalImagesPullingTime="42.416561058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:14.819440061 +0000 UTC m=+49.957717021" watchObservedRunningTime="2026-11-03 16:10:14.819878279 +0000 UTC m=+49.958155239"
Nov 03 16:10:15.559014 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:15.558979    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:15.559412 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:15.559047    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:15.573453 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:15.573432    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:16.400267 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.400235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:16.400435 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.400269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:16.400435 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:16.400346    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:16.400435 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:16.400421    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:16.400567 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.400448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:16.400567 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:16.400541    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:16.511094 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.511054    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7sv8"]
Nov 03 16:10:16.511755 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.511731    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cs98m"]
Nov 03 16:10:16.512923 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.512905    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qd8hz"]
Nov 03 16:10:16.561142 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.561111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:16.561558 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.561111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:16.561558 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:16.561243    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:16.561558 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:16.561120    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:16.561558 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:16.561297    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:16.561558 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:16.561382    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:18.399785 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:18.399601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:18.400262 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:18.399607    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:18.400262 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:18.399884    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:18.400262 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:18.399651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:18.400262 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:18.399946    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:18.400262 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:18.400002    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:20.399422 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:20.399395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:20.399941 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:20.399504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:20.399941 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:20.399512    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7sv8" podUID="2b0366db-bfff-4d9c-af6e-37b79c084739"
Nov 03 16:10:20.399941 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:20.399616    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cs98m" podUID="a16b1c0a-5c06-4fef-9a5a-664d92a24a25"
Nov 03 16:10:20.399941 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:20.399510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:20.399941 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:20.399712    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qd8hz" podUID="64a4a725-bb12-428a-a3d0-62622448f577"
Nov 03 16:10:22.174476 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.174451    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeReady"
Nov 03 16:10:22.174921 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.174587    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 16:10:22.286373 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.285292    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cff6v"]
Nov 03 16:10:22.288884 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.288864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cff6v"
Nov 03 16:10:22.323284 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.323258    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l9r4n"]
Nov 03 16:10:22.326570 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.326554    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"]
Nov 03 16:10:22.326696 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.326679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l9r4n"
Nov 03 16:10:22.330508 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.330486    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.331435 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.331419    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6l6l8"]
Nov 03 16:10:22.334640 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.334624    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-l75v9"]
Nov 03 16:10:22.334769 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.334753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6l6l8"
Nov 03 16:10:22.337727 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.337710    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-689b56979-4v24f"]
Nov 03 16:10:22.337824 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.337805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.340753 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.340732    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l2th2"]
Nov 03 16:10:22.340872 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.340856    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.343682 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.343665    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"]
Nov 03 16:10:22.343771 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.343762    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.346650 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.346631    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"]
Nov 03 16:10:22.346747 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.346740    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.349499 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.349481    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"]
Nov 03 16:10:22.349599 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.349565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.352481 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.352463    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"]
Nov 03 16:10:22.352626 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.352576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.355262 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.355248    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"
Nov 03 16:10:22.370814 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.370791    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"]
Nov 03 16:10:22.373948 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.373928    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-687cd8bf85-rh7wb"]
Nov 03 16:10:22.374053 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.374039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.378237 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.378217    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.378475 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.378454    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.379274 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.379253    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-z4krf"]
Nov 03 16:10:22.379461 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.379439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.382896 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.382874    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kqc54\""
Nov 03 16:10:22.382998 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.382969    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 16:10:22.383914 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.383577    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-65942"]
Nov 03 16:10:22.385366 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.385348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:22.389087 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.389064    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 16:10:22.391329 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.391310    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rwpz8"]
Nov 03 16:10:22.391446 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.391427    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942"
Nov 03 16:10:22.391613 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.391592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79s9t\" (UniqueName: \"kubernetes.io/projected/47333977-43b6-47af-ab13-b5dbee646be5-kube-api-access-79s9t\") pod \"kas-connection-checker-685895c7c9-cff6v\" (UID: \"47333977-43b6-47af-ab13-b5dbee646be5\") " pod="kube-system/kas-connection-checker-685895c7c9-cff6v"
Nov 03 16:10:22.395763 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.395274    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v4tnz"]
Nov 03 16:10:22.396385 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.396368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.399020 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399005    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cff6v"]
Nov 03 16:10:22.399099 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399031    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l9r4n"]
Nov 03 16:10:22.399099 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399047    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6l6l8"]
Nov 03 16:10:22.399099 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399065    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"]
Nov 03 16:10:22.399099 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399085    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-l75v9"]
Nov 03 16:10:22.399247 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399105    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-689b56979-4v24f"]
Nov 03 16:10:22.399247 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.399247 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399193    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:22.399479 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399443    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:22.399630 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.399595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:22.400995 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.400979    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"]
Nov 03 16:10:22.405194 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.405171    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"]
Nov 03 16:10:22.439475 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.439443    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-z4krf"]
Nov 03 16:10:22.445009 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.444969    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-65942"]
Nov 03 16:10:22.445147 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.445024    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwpz8"]
Nov 03 16:10:22.447481 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447284    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.447481 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447375    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 16:10:22.447933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447754    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 16:10:22.447933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447786    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.447933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447865    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.447933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447878    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.447933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447906    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.447933 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.447919    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.448288 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448003    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.448288 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448017    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.448288 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448039    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 16:10:22.448288 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448069    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.448288 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448102    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.448288 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448255    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 16:10:22.448577 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448519    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 16:10:22.448577 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448552    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.448577 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448567    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 16:10:22.448577 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448566    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.448766 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448749    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.448827 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448794    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 16:10:22.448878 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448835    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.448924 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448875    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 16:10:22.448924 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448879    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.448924 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.448897    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.449064 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.449013    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.449164 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.449132    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.449269 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.449254    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687cd8bf85-rh7wb"]
Nov 03 16:10:22.492615 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-service-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.492732 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-snapshots\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.492732 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.492732 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.492732 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.492904 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vrz\" (UniqueName: \"kubernetes.io/projected/36fe958f-eb20-4410-9e52-2a82273fce44-kube-api-access-q4vrz\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.492904 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.492976 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.492976 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.492967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khfm\" (UniqueName: \"kubernetes.io/projected/31c078e5-03f1-4b2e-ba52-5c10d58a5d82-kube-api-access-6khfm\") pod \"network-check-source-6f78d59f99-65942\" (UID: \"31c078e5-03f1-4b2e-ba52-5c10d58a5d82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942"
Nov 03 16:10:22.493046 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7351e46a-1bd0-430b-aec5-1f3111e80370-serving-cert\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.493080 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mht6z\" (UniqueName: \"kubernetes.io/projected/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-kube-api-access-mht6z\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.493125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.493169 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754fed8d-9c7a-4cee-8806-794ed3330bc0-tmp-dir\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.493169 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.493279 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-config\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.493279 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.493354 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.493354 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-cert\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.493354 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.493471 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7t6\" (UniqueName: \"kubernetes.io/projected/f7cd889c-edce-4208-9b4f-005d200d566b-kube-api-access-pv7t6\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.493471 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv75d\" (UniqueName: \"kubernetes.io/projected/7351e46a-1bd0-430b-aec5-1f3111e80370-kube-api-access-xv75d\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.493471 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:22.493601 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvcd\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.493601 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.493679 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsqfl\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.493679 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/11ba431e-e81b-4dbf-9d65-445a66ff6e07-kube-api-access-4w872\") pod \"kas-connection-checker-685895c7c9-l9r4n\" (UID: \"11ba431e-e81b-4dbf-9d65-445a66ff6e07\") " pod="kube-system/kas-connection-checker-685895c7c9-l9r4n"
Nov 03 16:10:22.493679 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.493777 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.493777 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.493777 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-default-certificate\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.493900 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7cd889c-edce-4208-9b4f-005d200d566b-serving-cert\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.493900 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.493900 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwrz8\" (UniqueName: \"kubernetes.io/projected/0dc284dd-fff4-439d-8732-7b78daf69085-kube-api-access-hwrz8\") pod \"kas-connection-checker-685895c7c9-6l6l8\" (UID: \"0dc284dd-fff4-439d-8732-7b78daf69085\") " pod="kube-system/kas-connection-checker-685895c7c9-6l6l8"
Nov 03 16:10:22.493978 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.493978 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-tmp\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.494042 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.493989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7jzc\" (UniqueName: \"kubernetes.io/projected/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-kube-api-access-k7jzc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.494042 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.494119 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.494119 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tv8f\" (UniqueName: \"kubernetes.io/projected/754fed8d-9c7a-4cee-8806-794ed3330bc0-kube-api-access-7tv8f\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.494232 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lrh\" (UniqueName: \"kubernetes.io/projected/d83906ba-ac2d-453a-b42b-3a3d958834ee-kube-api-access-w4lrh\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.494232 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494157    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.494232 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494194    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:22.494330 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtsj4\" (UniqueName: \"kubernetes.io/projected/290e7793-2a71-4dfb-9375-be6c7d26d7bb-kube-api-access-gtsj4\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.494330 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494297    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 16:10:22.494384 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.494412 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjrh\" (UniqueName: \"kubernetes.io/projected/be48d695-434e-473c-a500-a1e2e8c5de24-kube-api-access-wmjrh\") pod \"volume-data-source-validator-58c9f5f46d-kkwbq\" (UID: \"be48d695-434e-473c-a500-a1e2e8c5de24\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"
Nov 03 16:10:22.494445 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79s9t\" (UniqueName: \"kubernetes.io/projected/47333977-43b6-47af-ab13-b5dbee646be5-kube-api-access-79s9t\") pod \"kas-connection-checker-685895c7c9-cff6v\" (UID: \"47333977-43b6-47af-ab13-b5dbee646be5\") " pod="kube-system/kas-connection-checker-685895c7c9-cff6v"
Nov 03 16:10:22.494519 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7cd889c-edce-4208-9b4f-005d200d566b-config\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.494565 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/754fed8d-9c7a-4cee-8806-794ed3330bc0-config-volume\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.494565 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.494637 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-stats-auth\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.494637 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/290e7793-2a71-4dfb-9375-be6c7d26d7bb-serving-cert\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.494637 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.494757 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.494757 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtrrq\" (UniqueName: \"kubernetes.io/projected/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-kube-api-access-mtrrq\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.494842 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.494894 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.494865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.496021 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.495994    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 16:10:22.532584 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.532550    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 16:10:22.532691 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.532637    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 16:10:22.532847 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.532833    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 16:10:22.536291 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536276    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 16:10:22.536639 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536617    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 16:10:22.536745 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536656    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 16:10:22.536745 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536679    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 16:10:22.536745 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536702    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 16:10:22.536745 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536682    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 16:10:22.536911 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536756    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vtfh5\""
Nov 03 16:10:22.536911 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536677    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 16:10:22.536911 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536848    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 16:10:22.537025 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536923    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rqr4v\""
Nov 03 16:10:22.537025 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.536934    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 16:10:22.537478 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.537465    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"]
Nov 03 16:10:22.538032 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.538015    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"]
Nov 03 16:10:22.539508 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539492    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cqnft\""
Nov 03 16:10:22.539590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539529    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 16:10:22.539590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539540    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x47t4\""
Nov 03 16:10:22.539590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539565    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p6cn5\""
Nov 03 16:10:22.539590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539583    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dgbbj\""
Nov 03 16:10:22.539742 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539595    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7zlt\""
Nov 03 16:10:22.539742 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539651    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gswn6\""
Nov 03 16:10:22.539742 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539712    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zmbll\""
Nov 03 16:10:22.539842 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.539813    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 16:10:22.540034 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540019    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 16:10:22.540109 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540035    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:10:22.540109 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540027    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:10:22.540109 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540019    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zsc2l\""
Nov 03 16:10:22.540109 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540072    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mvkm5\""
Nov 03 16:10:22.540109 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540068    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-77nn9\""
Nov 03 16:10:22.540109 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540076    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 16:10:22.540109 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540101    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zz2z\""
Nov 03 16:10:22.540332 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540103    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 16:10:22.540332 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540144    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ggldv\""
Nov 03 16:10:22.540332 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540230    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hc4mx\""
Nov 03 16:10:22.540905 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.540891    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l2th2"]
Nov 03 16:10:22.541851 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.541837    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"]
Nov 03 16:10:22.542448 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.542433    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v4tnz"]
Nov 03 16:10:22.551668 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.551645    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79s9t\" (UniqueName: \"kubernetes.io/projected/47333977-43b6-47af-ab13-b5dbee646be5-kube-api-access-79s9t\") pod \"kas-connection-checker-685895c7c9-cff6v\" (UID: \"47333977-43b6-47af-ab13-b5dbee646be5\") " pod="kube-system/kas-connection-checker-685895c7c9-cff6v"
Nov 03 16:10:22.595758 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vrz\" (UniqueName: \"kubernetes.io/projected/36fe958f-eb20-4410-9e52-2a82273fce44-kube-api-access-q4vrz\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.595883 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.595883 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.595883 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khfm\" (UniqueName: \"kubernetes.io/projected/31c078e5-03f1-4b2e-ba52-5c10d58a5d82-kube-api-access-6khfm\") pod \"network-check-source-6f78d59f99-65942\" (UID: \"31c078e5-03f1-4b2e-ba52-5c10d58a5d82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942"
Nov 03 16:10:22.595883 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7351e46a-1bd0-430b-aec5-1f3111e80370-serving-cert\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.595883 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mht6z\" (UniqueName: \"kubernetes.io/projected/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-kube-api-access-mht6z\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.595891    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.595910    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687cd8bf85-rh7wb: secret "image-registry-tls" not found
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754fed8d-9c7a-4cee-8806-794ed3330bc0-tmp-dir\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.595939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.595962    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls podName:9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.09594474 +0000 UTC m=+58.234221693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls") pod "image-registry-687cd8bf85-rh7wb" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573") : secret "image-registry-tls" not found
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.596007    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-config\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.596053    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.596064    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls podName:d83906ba-ac2d-453a-b42b-3a3d958834ee nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.096038298 +0000 UTC m=+58.234315251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8vh2s" (UID: "d83906ba-ac2d-453a-b42b-3a3d958834ee") : secret "samples-operator-tls" not found
Nov 03 16:10:22.596125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.596140    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls podName:ef6cea18-3a39-4c4b-8fba-0284ca6f2e98 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.096108307 +0000 UTC m=+58.234385267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jnx8b" (UID: "ef6cea18-3a39-4c4b-8fba-0284ca6f2e98") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-cert\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7t6\" (UniqueName: \"kubernetes.io/projected/f7cd889c-edce-4208-9b4f-005d200d566b-kube-api-access-pv7t6\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv75d\" (UniqueName: \"kubernetes.io/projected/7351e46a-1bd0-430b-aec5-1f3111e80370-kube-api-access-xv75d\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvcd\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596458    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754fed8d-9c7a-4cee-8806-794ed3330bc0-tmp-dir\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsqfl\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/11ba431e-e81b-4dbf-9d65-445a66ff6e07-kube-api-access-4w872\") pod \"kas-connection-checker-685895c7c9-l9r4n\" (UID: \"11ba431e-e81b-4dbf-9d65-445a66ff6e07\") " pod="kube-system/kas-connection-checker-685895c7c9-l9r4n"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.596703 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-default-certificate\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7cd889c-edce-4208-9b4f-005d200d566b-serving-cert\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.596708    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.096691872 +0000 UTC m=+58.234968835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : configmap references non-existent config key: service-ca.crt
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwrz8\" (UniqueName: \"kubernetes.io/projected/0dc284dd-fff4-439d-8732-7b78daf69085-kube-api-access-hwrz8\") pod \"kas-connection-checker-685895c7c9-6l6l8\" (UID: \"0dc284dd-fff4-439d-8732-7b78daf69085\") " pod="kube-system/kas-connection-checker-685895c7c9-6l6l8"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.596804    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-tmp\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7jzc\" (UniqueName: \"kubernetes.io/projected/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-kube-api-access-k7jzc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596873    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-config\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.596946    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.596965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tv8f\" (UniqueName: \"kubernetes.io/projected/754fed8d-9c7a-4cee-8806-794ed3330bc0-kube-api-access-7tv8f\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lrh\" (UniqueName: \"kubernetes.io/projected/d83906ba-ac2d-453a-b42b-3a3d958834ee-kube-api-access-w4lrh\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.597019    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert podName:d671aa4a-c3a4-474d-8b2c-9b60ae49c0de nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.097000935 +0000 UTC m=+58.235277889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z4krf" (UID: "d671aa4a-c3a4-474d-8b2c-9b60ae49c0de") : secret "networking-console-plugin-cert" not found
Nov 03 16:10:22.598114 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.597038    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.597094    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls podName:754fed8d-9c7a-4cee-8806-794ed3330bc0 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.097082099 +0000 UTC m=+58.235359067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls") pod "dns-default-rwpz8" (UID: "754fed8d-9c7a-4cee-8806-794ed3330bc0") : secret "dns-default-metrics-tls" not found
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtsj4\" (UniqueName: \"kubernetes.io/projected/290e7793-2a71-4dfb-9375-be6c7d26d7bb-kube-api-access-gtsj4\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjrh\" (UniqueName: \"kubernetes.io/projected/be48d695-434e-473c-a500-a1e2e8c5de24-kube-api-access-wmjrh\") pod \"volume-data-source-validator-58c9f5f46d-kkwbq\" (UID: \"be48d695-434e-473c-a500-a1e2e8c5de24\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597255    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597275    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-tmp\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7cd889c-edce-4208-9b4f-005d200d566b-config\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.597328    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.09731222 +0000 UTC m=+58.235589162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : secret "router-metrics-certs-default" not found
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/754fed8d-9c7a-4cee-8806-794ed3330bc0-config-volume\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-stats-auth\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/290e7793-2a71-4dfb-9375-be6c7d26d7bb-serving-cert\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.598870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtrrq\" (UniqueName: \"kubernetes.io/projected/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-kube-api-access-mtrrq\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-service-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-snapshots\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597851    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.598494    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.598496    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7cd889c-edce-4208-9b4f-005d200d566b-config\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.598969    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-service-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599068    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-snapshots\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599167    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.599590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599266    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/754fed8d-9c7a-4cee-8806-794ed3330bc0-config-volume\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599335    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599533    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599665    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599833    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-cert\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.599996    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.600088    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.600102    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h: secret "image-registry-tls" not found
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:22.600147    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls podName:9cf495d7-831a-4266-a508-c12d9968971d nodeName:}" failed. No retries permitted until 2026-11-03 16:10:23.100127893 +0000 UTC m=+58.238404854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls") pod "image-registry-6f4bdbf4dd-bqs7h" (UID: "9cf495d7-831a-4266-a508-c12d9968971d") : secret "image-registry-tls" not found
Nov 03 16:10:22.600383 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.597717    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.600810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.600404    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.600810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.600443    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7351e46a-1bd0-430b-aec5-1f3111e80370-serving-cert\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.600810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.600540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.600810 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.600759    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.601381 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.601359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cff6v"
Nov 03 16:10:22.602363 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.602340    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.603354 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.603097    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.603354 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.603264    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.603354 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.603313    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.603995 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.603971    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7cd889c-edce-4208-9b4f-005d200d566b-serving-cert\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.603995 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.603986    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-default-certificate\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.604261 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.604237    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/290e7793-2a71-4dfb-9375-be6c7d26d7bb-serving-cert\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.604564 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.604548    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-stats-auth\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.604775 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.604759    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.654032    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv75d\" (UniqueName: \"kubernetes.io/projected/7351e46a-1bd0-430b-aec5-1f3111e80370-kube-api-access-xv75d\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.654307    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khfm\" (UniqueName: \"kubernetes.io/projected/31c078e5-03f1-4b2e-ba52-5c10d58a5d82-kube-api-access-6khfm\") pod \"network-check-source-6f78d59f99-65942\" (UID: \"31c078e5-03f1-4b2e-ba52-5c10d58a5d82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.654919    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mht6z\" (UniqueName: \"kubernetes.io/projected/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-kube-api-access-mht6z\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.655318    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7t6\" (UniqueName: \"kubernetes.io/projected/f7cd889c-edce-4208-9b4f-005d200d566b-kube-api-access-pv7t6\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.656495    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.656873    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtsj4\" (UniqueName: \"kubernetes.io/projected/290e7793-2a71-4dfb-9375-be6c7d26d7bb-kube-api-access-gtsj4\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.657500    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsqfl\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.658306 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.657535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:22.658815 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.658528    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7jzc\" (UniqueName: \"kubernetes.io/projected/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-kube-api-access-k7jzc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.659019 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.658970    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwrz8\" (UniqueName: \"kubernetes.io/projected/0dc284dd-fff4-439d-8732-7b78daf69085-kube-api-access-hwrz8\") pod \"kas-connection-checker-685895c7c9-6l6l8\" (UID: \"0dc284dd-fff4-439d-8732-7b78daf69085\") " pod="kube-system/kas-connection-checker-685895c7c9-6l6l8"
Nov 03 16:10:22.660370 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.660018    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tv8f\" (UniqueName: \"kubernetes.io/projected/754fed8d-9c7a-4cee-8806-794ed3330bc0-kube-api-access-7tv8f\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:22.660370 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.660311    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvcd\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:22.663801 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.661771    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vrz\" (UniqueName: \"kubernetes.io/projected/36fe958f-eb20-4410-9e52-2a82273fce44-kube-api-access-q4vrz\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:22.664745 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.664128    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lrh\" (UniqueName: \"kubernetes.io/projected/d83906ba-ac2d-453a-b42b-3a3d958834ee-kube-api-access-w4lrh\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:22.664745 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.664567    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtrrq\" (UniqueName: \"kubernetes.io/projected/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-kube-api-access-mtrrq\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:22.665785 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.665741    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjrh\" (UniqueName: \"kubernetes.io/projected/be48d695-434e-473c-a500-a1e2e8c5de24-kube-api-access-wmjrh\") pod \"volume-data-source-validator-58c9f5f46d-kkwbq\" (UID: \"be48d695-434e-473c-a500-a1e2e8c5de24\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"
Nov 03 16:10:22.665860 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.665835    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/11ba431e-e81b-4dbf-9d65-445a66ff6e07-kube-api-access-4w872\") pod \"kas-connection-checker-685895c7c9-l9r4n\" (UID: \"11ba431e-e81b-4dbf-9d65-445a66ff6e07\") " pod="kube-system/kas-connection-checker-685895c7c9-l9r4n"
Nov 03 16:10:22.666075 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.666053    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:22.669138 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.669112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:22.684367 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.684335    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:22.692600 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.692573    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:22.701320 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.701292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"
Nov 03 16:10:22.728695 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.727542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942"
Nov 03 16:10:22.737988 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.737800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:22.939466 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.939401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l9r4n"
Nov 03 16:10:22.952451 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.952427    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6l6l8"
Nov 03 16:10:22.983439 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:22.983417    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cff6v"]
Nov 03 16:10:22.986298 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:22.986273    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47333977_43b6_47af_ab13_b5dbee646be5.slice/crio-8cfdaeb4197a9a33728c70254a2094bad01b8bec4ea357f4c752fe1061fa0914 WatchSource:0}: Error finding container 8cfdaeb4197a9a33728c70254a2094bad01b8bec4ea357f4c752fe1061fa0914: Status 404 returned error can't find the container with id 8cfdaeb4197a9a33728c70254a2094bad01b8bec4ea357f4c752fe1061fa0914
Nov 03 16:10:23.036581 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.036439    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-l75v9"]
Nov 03 16:10:23.066037 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.066006    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l2th2"]
Nov 03 16:10:23.069303 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:23.069275    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290e7793_2a71_4dfb_9375_be6c7d26d7bb.slice/crio-71467ccfa2ab2d54da1b774f3abb5223d2a7140599b31ef85597b28a347b3108 WatchSource:0}: Error finding container 71467ccfa2ab2d54da1b774f3abb5223d2a7140599b31ef85597b28a347b3108: Status 404 returned error can't find the container with id 71467ccfa2ab2d54da1b774f3abb5223d2a7140599b31ef85597b28a347b3108
Nov 03 16:10:23.073839 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.073816    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"]
Nov 03 16:10:23.083927 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.083908    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"]
Nov 03 16:10:23.103873 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.103849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:23.103951 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.103883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:23.104006 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.103971    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:10:23.104006 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.103972    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:10:23.104077 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104009    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.103995113 +0000 UTC m=+59.242272053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : secret "router-metrics-certs-default" not found
Nov 03 16:10:23.104077 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.104048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:23.104148 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104078    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls podName:754fed8d-9c7a-4cee-8806-794ed3330bc0 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.104064939 +0000 UTC m=+59.242341878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls") pod "dns-default-rwpz8" (UID: "754fed8d-9c7a-4cee-8806-794ed3330bc0") : secret "dns-default-metrics-tls" not found
Nov 03 16:10:23.104148 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.104096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:23.104148 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104102    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:23.104148 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104111    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h: secret "image-registry-tls" not found
Nov 03 16:10:23.104148 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.104129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:23.104148 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104132    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls podName:9cf495d7-831a-4266-a508-c12d9968971d nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.104125795 +0000 UTC m=+59.242402735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls") pod "image-registry-6f4bdbf4dd-bqs7h" (UID: "9cf495d7-831a-4266-a508-c12d9968971d") : secret "image-registry-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.104155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104166    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104172    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687cd8bf85-rh7wb: secret "image-registry-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.104176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104191    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls podName:9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.104184126 +0000 UTC m=+59.242461064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls") pod "image-registry-687cd8bf85-rh7wb" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573") : secret "image-registry-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.104248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104262    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104278    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.104267577 +0000 UTC m=+59.242544518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : configmap references non-existent config key: service-ca.crt
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104287    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104300    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls podName:d83906ba-ac2d-453a-b42b-3a3d958834ee nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.10429005 +0000 UTC m=+59.242566990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8vh2s" (UID: "d83906ba-ac2d-453a-b42b-3a3d958834ee") : secret "samples-operator-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104321    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104341    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls podName:ef6cea18-3a39-4c4b-8fba-0284ca6f2e98 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.10432505 +0000 UTC m=+59.242602002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jnx8b" (UID: "ef6cea18-3a39-4c4b-8fba-0284ca6f2e98") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:23.104409 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:23.104363    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert podName:d671aa4a-c3a4-474d-8b2c-9b60ae49c0de nodeName:}" failed. No retries permitted until 2026-11-03 16:10:24.104352586 +0000 UTC m=+59.242629525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z4krf" (UID: "d671aa4a-c3a4-474d-8b2c-9b60ae49c0de") : secret "networking-console-plugin-cert" not found
Nov 03 16:10:23.153808 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.153763    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"]
Nov 03 16:10:23.154842 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.154782    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v4tnz"]
Nov 03 16:10:23.155836 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.155815    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-65942"]
Nov 03 16:10:23.157165 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:23.157132    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe48d695_434e_473c_a500_a1e2e8c5de24.slice/crio-6c71a4b665035f72770ba76a9ac8a6d4e348e8a19a6437e89fc09ce5135725fb WatchSource:0}: Error finding container 6c71a4b665035f72770ba76a9ac8a6d4e348e8a19a6437e89fc09ce5135725fb: Status 404 returned error can't find the container with id 6c71a4b665035f72770ba76a9ac8a6d4e348e8a19a6437e89fc09ce5135725fb
Nov 03 16:10:23.157732 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:23.157706    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c078e5_03f1_4b2e_ba52_5c10d58a5d82.slice/crio-814f0b4743c0ec6d09533d85123858f6e90b9e9ee04bd96edb8e3544da3e60e3 WatchSource:0}: Error finding container 814f0b4743c0ec6d09533d85123858f6e90b9e9ee04bd96edb8e3544da3e60e3: Status 404 returned error can't find the container with id 814f0b4743c0ec6d09533d85123858f6e90b9e9ee04bd96edb8e3544da3e60e3
Nov 03 16:10:23.158408 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:23.158386    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75bf77f_4e2f_43d3_aa27_1ee258dc4db6.slice/crio-30468e0b0f5f72ffe3f2e9a8513b606f743f8bab828a3e2eceac5555d535d23a WatchSource:0}: Error finding container 30468e0b0f5f72ffe3f2e9a8513b606f743f8bab828a3e2eceac5555d535d23a: Status 404 returned error can't find the container with id 30468e0b0f5f72ffe3f2e9a8513b606f743f8bab828a3e2eceac5555d535d23a
Nov 03 16:10:23.309040 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.308968    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l9r4n"]
Nov 03 16:10:23.311706 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:23.311672    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ba431e_e81b_4dbf_9d65_445a66ff6e07.slice/crio-8666150bf8975e316066a5130ded1b551fb1808b983e7906a4c97f3b1f54fa25 WatchSource:0}: Error finding container 8666150bf8975e316066a5130ded1b551fb1808b983e7906a4c97f3b1f54fa25: Status 404 returned error can't find the container with id 8666150bf8975e316066a5130ded1b551fb1808b983e7906a4c97f3b1f54fa25
Nov 03 16:10:23.335735 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.335707    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6l6l8"]
Nov 03 16:10:23.339357 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:23.339334    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc284dd_fff4_439d_8732_7b78daf69085.slice/crio-7dd70c2f054731ac3591729fa400fe23995450439706c52b87b7d22e4933c329 WatchSource:0}: Error finding container 7dd70c2f054731ac3591729fa400fe23995450439706c52b87b7d22e4933c329: Status 404 returned error can't find the container with id 7dd70c2f054731ac3591729fa400fe23995450439706c52b87b7d22e4933c329
Nov 03 16:10:23.577955 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.577866    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l9r4n" event={"ID":"11ba431e-e81b-4dbf-9d65-445a66ff6e07","Type":"ContainerStarted","Data":"fe17e6682b703eb17e7bc99c8daa21034aae01e5f066aaf99ada0cadaba94c9e"}
Nov 03 16:10:23.577955 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.577929    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l9r4n" event={"ID":"11ba431e-e81b-4dbf-9d65-445a66ff6e07","Type":"ContainerStarted","Data":"8666150bf8975e316066a5130ded1b551fb1808b983e7906a4c97f3b1f54fa25"}
Nov 03 16:10:23.579036 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.579010    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v4tnz" event={"ID":"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6","Type":"ContainerStarted","Data":"30468e0b0f5f72ffe3f2e9a8513b606f743f8bab828a3e2eceac5555d535d23a"}
Nov 03 16:10:23.580061 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.580039    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2" event={"ID":"290e7793-2a71-4dfb-9375-be6c7d26d7bb","Type":"ContainerStarted","Data":"71467ccfa2ab2d54da1b774f3abb5223d2a7140599b31ef85597b28a347b3108"}
Nov 03 16:10:23.581434 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.581410    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6l6l8" event={"ID":"0dc284dd-fff4-439d-8732-7b78daf69085","Type":"ContainerStarted","Data":"13581463571cb2abef2b627fe3e62e1cf055b708455973c4ba111497b5226a4e"}
Nov 03 16:10:23.581519 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.581448    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6l6l8" event={"ID":"0dc284dd-fff4-439d-8732-7b78daf69085","Type":"ContainerStarted","Data":"7dd70c2f054731ac3591729fa400fe23995450439706c52b87b7d22e4933c329"}
Nov 03 16:10:23.582528 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.582499    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942" event={"ID":"31c078e5-03f1-4b2e-ba52-5c10d58a5d82","Type":"ContainerStarted","Data":"814f0b4743c0ec6d09533d85123858f6e90b9e9ee04bd96edb8e3544da3e60e3"}
Nov 03 16:10:23.583522 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.583501    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b" event={"ID":"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0","Type":"ContainerStarted","Data":"20d25fae91f0b396d21563561efdfd1b0e72e5696f63ad195d5f21918944a802"}
Nov 03 16:10:23.584787 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.584761    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l75v9" event={"ID":"7351e46a-1bd0-430b-aec5-1f3111e80370","Type":"ContainerStarted","Data":"231d0cf57d8714e43577c44dbdb66bd96cf2040e27b71c48ae813c5166f4d77b"}
Nov 03 16:10:23.586426 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.586400    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cff6v" event={"ID":"47333977-43b6-47af-ab13-b5dbee646be5","Type":"ContainerStarted","Data":"dc454a0e09983d85b40fc313b5a9b19ae0c4ed347de12847b12eb3a4567422d9"}
Nov 03 16:10:23.586556 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.586448    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cff6v" event={"ID":"47333977-43b6-47af-ab13-b5dbee646be5","Type":"ContainerStarted","Data":"8cfdaeb4197a9a33728c70254a2094bad01b8bec4ea357f4c752fe1061fa0914"}
Nov 03 16:10:23.588090 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.588062    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq" event={"ID":"be48d695-434e-473c-a500-a1e2e8c5de24","Type":"ContainerStarted","Data":"6c71a4b665035f72770ba76a9ac8a6d4e348e8a19a6437e89fc09ce5135725fb"}
Nov 03 16:10:23.589252 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:23.589234    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh" event={"ID":"f7cd889c-edce-4208-9b4f-005d200d566b","Type":"ContainerStarted","Data":"3d34a9f85672264c2ee4b2649d9e8db0810adbff1a134594fb3f21b1dbf9c524"}
Nov 03 16:10:24.113730 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.113694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:24.113918 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.113787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:24.113918 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.113832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:24.113918 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.113868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:24.114075 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.113901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:24.114075 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.113965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:24.114075 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.114003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:24.114075 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.114035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:24.114295 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114274    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:10:24.114345 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114338    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.114319577 +0000 UTC m=+61.252596518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : secret "router-metrics-certs-default" not found
Nov 03 16:10:24.114431 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114418    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:24.114480 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114435    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h: secret "image-registry-tls" not found
Nov 03 16:10:24.114524 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114469    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls podName:9cf495d7-831a-4266-a508-c12d9968971d nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.114457952 +0000 UTC m=+61.252735089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls") pod "image-registry-6f4bdbf4dd-bqs7h" (UID: "9cf495d7-831a-4266-a508-c12d9968971d") : secret "image-registry-tls" not found
Nov 03 16:10:24.114615 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114602    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:24.114664 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114618    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687cd8bf85-rh7wb: secret "image-registry-tls" not found
Nov 03 16:10:24.114664 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114653    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls podName:9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.114641594 +0000 UTC m=+61.252918536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls") pod "image-registry-687cd8bf85-rh7wb" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573") : secret "image-registry-tls" not found
Nov 03 16:10:24.114762 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114725    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.114714088 +0000 UTC m=+61.252991029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : configmap references non-existent config key: service-ca.crt
Nov 03 16:10:24.114875 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114862    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:24.114929 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114905    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls podName:ef6cea18-3a39-4c4b-8fba-0284ca6f2e98 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.114891969 +0000 UTC m=+61.253168917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jnx8b" (UID: "ef6cea18-3a39-4c4b-8fba-0284ca6f2e98") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:24.115002 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.114969    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:10:24.115046 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.115028    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls podName:d83906ba-ac2d-453a-b42b-3a3d958834ee nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.115017179 +0000 UTC m=+61.253294125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8vh2s" (UID: "d83906ba-ac2d-453a-b42b-3a3d958834ee") : secret "samples-operator-tls" not found
Nov 03 16:10:24.115098 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.115086    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:10:24.115131 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.115123    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert podName:d671aa4a-c3a4-474d-8b2c-9b60ae49c0de nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.115112406 +0000 UTC m=+61.253389348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z4krf" (UID: "d671aa4a-c3a4-474d-8b2c-9b60ae49c0de") : secret "networking-console-plugin-cert" not found
Nov 03 16:10:24.115206 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.115182    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:10:24.115316 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:24.115304    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls podName:754fed8d-9c7a-4cee-8806-794ed3330bc0 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:26.115292593 +0000 UTC m=+61.253569541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls") pod "dns-default-rwpz8" (UID: "754fed8d-9c7a-4cee-8806-794ed3330bc0") : secret "dns-default-metrics-tls" not found
Nov 03 16:10:24.406243 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.406167    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cff6v" podStartSLOduration=393.406148452 podStartE2EDuration="6m33.406148452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:24.389061588 +0000 UTC m=+59.527338551" watchObservedRunningTime="2026-11-03 16:10:24.406148452 +0000 UTC m=+59.544425414"
Nov 03 16:10:24.631973 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.631912    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l9r4n" podStartSLOduration=393.631892214 podStartE2EDuration="6m33.631892214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:24.518992291 +0000 UTC m=+59.657269252" watchObservedRunningTime="2026-11-03 16:10:24.631892214 +0000 UTC m=+59.770169175"
Nov 03 16:10:24.632225 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:24.632011    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6l6l8" podStartSLOduration=393.632006192 podStartE2EDuration="6m33.632006192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:24.630656793 +0000 UTC m=+59.768933751" watchObservedRunningTime="2026-11-03 16:10:24.632006192 +0000 UTC m=+59.770283154"
Nov 03 16:10:26.136779 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.136736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.136794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.136885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.136918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.136959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.136995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.137039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.137082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:26.137289 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137262    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:10:26.137655 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137328    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert podName:d671aa4a-c3a4-474d-8b2c-9b60ae49c0de nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.137309252 +0000 UTC m=+65.275586207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z4krf" (UID: "d671aa4a-c3a4-474d-8b2c-9b60ae49c0de") : secret "networking-console-plugin-cert" not found
Nov 03 16:10:26.137763 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137743    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:10:26.137845 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137802    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls podName:754fed8d-9c7a-4cee-8806-794ed3330bc0 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.137785777 +0000 UTC m=+65.276062732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls") pod "dns-default-rwpz8" (UID: "754fed8d-9c7a-4cee-8806-794ed3330bc0") : secret "dns-default-metrics-tls" not found
Nov 03 16:10:26.137903 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137871    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:10:26.137965 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137905    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.137894257 +0000 UTC m=+65.276171213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : secret "router-metrics-certs-default" not found
Nov 03 16:10:26.138030 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137967    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:26.138030 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.137979    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h: secret "image-registry-tls" not found
Nov 03 16:10:26.138030 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138014    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls podName:9cf495d7-831a-4266-a508-c12d9968971d nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.13800289 +0000 UTC m=+65.276279831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls") pod "image-registry-6f4bdbf4dd-bqs7h" (UID: "9cf495d7-831a-4266-a508-c12d9968971d") : secret "image-registry-tls" not found
Nov 03 16:10:26.138182 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138075    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:26.138182 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138085    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687cd8bf85-rh7wb: secret "image-registry-tls" not found
Nov 03 16:10:26.138182 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138114    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls podName:9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.138103108 +0000 UTC m=+65.276380050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls") pod "image-registry-687cd8bf85-rh7wb" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573") : secret "image-registry-tls" not found
Nov 03 16:10:26.138364 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138190    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.138179603 +0000 UTC m=+65.276456556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : configmap references non-existent config key: service-ca.crt
Nov 03 16:10:26.138364 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138270    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:26.138364 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138312    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls podName:ef6cea18-3a39-4c4b-8fba-0284ca6f2e98 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.13830137 +0000 UTC m=+65.276578313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jnx8b" (UID: "ef6cea18-3a39-4c4b-8fba-0284ca6f2e98") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:26.138504 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138356    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:10:26.138504 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:26.138435    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls podName:d83906ba-ac2d-453a-b42b-3a3d958834ee nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.138415546 +0000 UTC m=+65.276692514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8vh2s" (UID: "d83906ba-ac2d-453a-b42b-3a3d958834ee") : secret "samples-operator-tls" not found
Nov 03 16:10:26.607439 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.607394    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq" event={"ID":"be48d695-434e-473c-a500-a1e2e8c5de24","Type":"ContainerStarted","Data":"421124409f288b1c6d371b56dd4896571b52c7c0cc1a4d230352e26b7a31d5cd"}
Nov 03 16:10:26.718988 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:26.718907    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq" podStartSLOduration=362.166326499 podStartE2EDuration="6m4.718893698s" totalImagesPullingTime="2.552567199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:26.717731659 +0000 UTC m=+61.856008625" watchObservedRunningTime="2026-11-03 16:10:26.718893698 +0000 UTC m=+61.857170653"
Nov 03 16:10:27.614454 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:27.613630    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l75v9" event={"ID":"7351e46a-1bd0-430b-aec5-1f3111e80370","Type":"ContainerStarted","Data":"6f6b03577588e47dcf604ce92d13f39c41ca03d2f648fd8d0cff7f94c6508e2f"}
Nov 03 16:10:27.617152 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:27.617129    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/0.log"
Nov 03 16:10:27.617305 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:27.617163    2582 generic.go:356] "Generic (PLEG): container finished" podID="290e7793-2a71-4dfb-9375-be6c7d26d7bb" containerID="8b4a4978c4dca9f272897c37405b8e84f997f6c716a4c2dffc8277644dd49d60" exitCode=255
Nov 03 16:10:27.617965 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:27.617403    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2" event={"ID":"290e7793-2a71-4dfb-9375-be6c7d26d7bb","Type":"ContainerDied","Data":"8b4a4978c4dca9f272897c37405b8e84f997f6c716a4c2dffc8277644dd49d60"}
Nov 03 16:10:27.617965 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:27.617667    2582 scope.go:122] "RemoveContainer" containerID="8b4a4978c4dca9f272897c37405b8e84f997f6c716a4c2dffc8277644dd49d60"
Nov 03 16:10:27.731867 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:27.730609    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-l75v9" podStartSLOduration=361.74937127 podStartE2EDuration="6m5.730589561s" totalImagesPullingTime="3.981218291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:27.729561416 +0000 UTC m=+62.867838605" watchObservedRunningTime="2026-11-03 16:10:27.730589561 +0000 UTC m=+62.868866524"
Nov 03 16:10:28.624255 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.623230    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b" event={"ID":"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0","Type":"ContainerStarted","Data":"20a0b4c77827b92d95adf0b7981eaf0959c245e637930dbdc85590f649aeeba3"}
Nov 03 16:10:28.625014 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.624983    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh" event={"ID":"f7cd889c-edce-4208-9b4f-005d200d566b","Type":"ContainerStarted","Data":"23c35647f028e8675616f632e43811e95bed9a5501db28165a3b3d8e91c5b87d"}
Nov 03 16:10:28.626514 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.626485    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v4tnz" event={"ID":"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6","Type":"ContainerStarted","Data":"9eba3c9ae0d25825bffae900f577b790e7608fb6ce53b6349b7b8e212c31fef3"}
Nov 03 16:10:28.628042 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.628025    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:10:28.628461 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.628445    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/0.log"
Nov 03 16:10:28.628533 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.628482    2582 generic.go:356] "Generic (PLEG): container finished" podID="290e7793-2a71-4dfb-9375-be6c7d26d7bb" containerID="5baeab839e0d08d18a7338066961f762077bc9655868802f6ab04a19c2afbe98" exitCode=255
Nov 03 16:10:28.628590 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.628551    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2" event={"ID":"290e7793-2a71-4dfb-9375-be6c7d26d7bb","Type":"ContainerDied","Data":"5baeab839e0d08d18a7338066961f762077bc9655868802f6ab04a19c2afbe98"}
Nov 03 16:10:28.628623 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.628587    2582 scope.go:122] "RemoveContainer" containerID="8b4a4978c4dca9f272897c37405b8e84f997f6c716a4c2dffc8277644dd49d60"
Nov 03 16:10:28.628888 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.628872    2582 scope.go:122] "RemoveContainer" containerID="5baeab839e0d08d18a7338066961f762077bc9655868802f6ab04a19c2afbe98"
Nov 03 16:10:28.629082 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:28.629060    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l2th2_openshift-console-operator(290e7793-2a71-4dfb-9375-be6c7d26d7bb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2" podUID="290e7793-2a71-4dfb-9375-be6c7d26d7bb"
Nov 03 16:10:28.683880 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.683855    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8szwq"]
Nov 03 16:10:28.688309 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.688291    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.736672 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.736627    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b" podStartSLOduration=361.928904553 podStartE2EDuration="6m6.736612127s" totalImagesPullingTime="4.807707574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:28.735732471 +0000 UTC m=+63.874009432" watchObservedRunningTime="2026-11-03 16:10:28.736612127 +0000 UTC m=+63.874889120"
Nov 03 16:10:28.739723 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.739700    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 16:10:28.739823 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.739803    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 16:10:28.739866 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.739839    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4pfhl\""
Nov 03 16:10:28.744279 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.744257    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8szwq"]
Nov 03 16:10:28.767298 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.767275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a241b6-335b-4ee8-add5-07a5e71666e5-crio-socket\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.767416 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.767357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.767416 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.767389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.767554 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.767538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a241b6-335b-4ee8-add5-07a5e71666e5-data-volume\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.767660 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.767640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bwg\" (UniqueName: \"kubernetes.io/projected/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-api-access-h6bwg\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.849729 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.849671    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh" podStartSLOduration=361.710473286 podStartE2EDuration="6m6.849654469s" totalImagesPullingTime="5.139181183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:28.848886481 +0000 UTC m=+63.987163441" watchObservedRunningTime="2026-11-03 16:10:28.849654469 +0000 UTC m=+63.987931432"
Nov 03 16:10:28.868796 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.868762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bwg\" (UniqueName: \"kubernetes.io/projected/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-api-access-h6bwg\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.868968 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.868869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a241b6-335b-4ee8-add5-07a5e71666e5-crio-socket\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.869023 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.868977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.869023 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.869008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.869120 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.869070    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a241b6-335b-4ee8-add5-07a5e71666e5-crio-socket\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.869182 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.869165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a241b6-335b-4ee8-add5-07a5e71666e5-data-volume\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.869533 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:28.869175    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:28.869533 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:28.869361    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:29.369339149 +0000 UTC m=+64.507616102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:28.869533 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:28.869189    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:28.869533 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:28.869439    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:29.369420288 +0000 UTC m=+64.507697234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:28.869533 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.869488    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a241b6-335b-4ee8-add5-07a5e71666e5-data-volume\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.923763 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.923734    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xxtp9-hcmjv"]
Nov 03 16:10:28.927052 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.927031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:28.928663 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.928639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bwg\" (UniqueName: \"kubernetes.io/projected/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-api-access-h6bwg\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:28.969870 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.969841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:28.969973 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.969914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:28.970020 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.969968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjk5\" (UniqueName: \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:28.970137 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.970122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:28.982816 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:28.982784    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xxtp9-hcmjv"]
Nov 03 16:10:29.071117 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.071072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.071117 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.071119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjk5\" (UniqueName: \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.071409 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.071312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.071841 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.071555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.072017 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.071988    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.072349 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.072332    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.072389 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.072335    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.072389 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.072347    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v4tnz" podStartSLOduration=2.080181787 podStartE2EDuration="7.072339487s" totalImagesPullingTime="4.9921577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:29.071937819 +0000 UTC m=+64.210214785" watchObservedRunningTime="2026-11-03 16:10:29.072339487 +0000 UTC m=+64.210616449"
Nov 03 16:10:29.127675 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.127643    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjk5\" (UniqueName: \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.245232 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.245142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:29.374388 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.374356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:29.374388 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.374389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:29.374563 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:29.374476    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:29.374563 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:29.374535    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.374519787 +0000 UTC m=+65.512796727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:29.374652 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:29.374580    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:29.374652 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:29.374621    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:30.374611095 +0000 UTC m=+65.512888048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:29.633642 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.633566    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:10:29.634021 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.633981    2582 scope.go:122] "RemoveContainer" containerID="5baeab839e0d08d18a7338066961f762077bc9655868802f6ab04a19c2afbe98"
Nov 03 16:10:29.634251 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:29.634221    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l2th2_openshift-console-operator(290e7793-2a71-4dfb-9375-be6c7d26d7bb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2" podUID="290e7793-2a71-4dfb-9375-be6c7d26d7bb"
Nov 03 16:10:29.635327 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.635292    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942" event={"ID":"31c078e5-03f1-4b2e-ba52-5c10d58a5d82","Type":"ContainerStarted","Data":"3b9e4871c357246ea9178f3111062a80b6d2a3bcd39de2a9532cba1fc3646672"}
Nov 03 16:10:29.648447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.648422    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"]
Nov 03 16:10:29.651242 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:29.651221    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad56a4f_f2b5_4dae_8bdb_36a008d1645e.slice/crio-4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779 WatchSource:0}: Error finding container 4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779: Status 404 returned error can't find the container with id 4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779
Nov 03 16:10:29.652885 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.652871    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xxtp9-hcmjv"]
Nov 03 16:10:29.652955 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.652944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"
Nov 03 16:10:29.660262 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.660241    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"]
Nov 03 16:10:29.664786 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.664773    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"
Nov 03 16:10:29.677249 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.677192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4hz\" (UniqueName: \"kubernetes.io/projected/b5c0882b-45cb-47a7-983e-01e5748ffab9-kube-api-access-zg4hz\") pod \"migrator-66df5d496b-svr6p\" (UID: \"b5c0882b-45cb-47a7-983e-01e5748ffab9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"
Nov 03 16:10:29.677411 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.677395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp5xr\" (UniqueName: \"kubernetes.io/projected/447a7d8a-1272-419d-a815-6b4353a2011a-kube-api-access-tp5xr\") pod \"migrator-66df5d496b-2b5rh\" (UID: \"447a7d8a-1272-419d-a815-6b4353a2011a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"
Nov 03 16:10:29.709402 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.709377    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 16:10:29.711565 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.711530    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"]
Nov 03 16:10:29.772746 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.772714    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"]
Nov 03 16:10:29.778373 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.778345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5xr\" (UniqueName: \"kubernetes.io/projected/447a7d8a-1272-419d-a815-6b4353a2011a-kube-api-access-tp5xr\") pod \"migrator-66df5d496b-2b5rh\" (UID: \"447a7d8a-1272-419d-a815-6b4353a2011a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"
Nov 03 16:10:29.778581 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.778559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4hz\" (UniqueName: \"kubernetes.io/projected/b5c0882b-45cb-47a7-983e-01e5748ffab9-kube-api-access-zg4hz\") pod \"migrator-66df5d496b-svr6p\" (UID: \"b5c0882b-45cb-47a7-983e-01e5748ffab9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"
Nov 03 16:10:29.778646 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.778576    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wnjzm\""
Nov 03 16:10:29.807441 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.807410    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:29.834607 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.834579    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5xr\" (UniqueName: \"kubernetes.io/projected/447a7d8a-1272-419d-a815-6b4353a2011a-kube-api-access-tp5xr\") pod \"migrator-66df5d496b-2b5rh\" (UID: \"447a7d8a-1272-419d-a815-6b4353a2011a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"
Nov 03 16:10:29.834744 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.834625    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4hz\" (UniqueName: \"kubernetes.io/projected/b5c0882b-45cb-47a7-983e-01e5748ffab9-kube-api-access-zg4hz\") pod \"migrator-66df5d496b-svr6p\" (UID: \"b5c0882b-45cb-47a7-983e-01e5748ffab9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"
Nov 03 16:10:29.967757 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.967727    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"
Nov 03 16:10:29.974572 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.974527    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942" podStartSLOduration=359.474742795 podStartE2EDuration="6m4.974513067s" totalImagesPullingTime="5.499770272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:29.973173144 +0000 UTC m=+65.111450107" watchObservedRunningTime="2026-11-03 16:10:29.974513067 +0000 UTC m=+65.112790027"
Nov 03 16:10:29.989845 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:29.989820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"
Nov 03 16:10:30.182508 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.182474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:30.182508 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.182509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:30.182696 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182605    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 16:10:30.182696 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182612    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 16:10:30.182696 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182665    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.182651698 +0000 UTC m=+73.320928651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : secret "router-metrics-certs-default" not found
Nov 03 16:10:30.182791 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182738    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls podName:754fed8d-9c7a-4cee-8806-794ed3330bc0 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.182722762 +0000 UTC m=+73.320999702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls") pod "dns-default-rwpz8" (UID: "754fed8d-9c7a-4cee-8806-794ed3330bc0") : secret "dns-default-metrics-tls" not found
Nov 03 16:10:30.182867 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.182791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:30.182867 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.182815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:30.182867 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.182841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:30.182986 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.182867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:30.182986 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.182895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:30.182986 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182921    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:30.182986 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182934    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687cd8bf85-rh7wb: secret "image-registry-tls" not found
Nov 03 16:10:30.182986 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182971    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls podName:9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.182956816 +0000 UTC m=+73.321233754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls") pod "image-registry-687cd8bf85-rh7wb" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573") : secret "image-registry-tls" not found
Nov 03 16:10:30.182986 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182971    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 16:10:30.182986 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182921    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182993    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h: secret "image-registry-tls" not found
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.182998    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.183003    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle podName:36fe958f-eb20-4410-9e52-2a82273fce44 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.182980688 +0000 UTC m=+73.321257654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle") pod "router-default-689b56979-4v24f" (UID: "36fe958f-eb20-4410-9e52-2a82273fce44") : configmap references non-existent config key: service-ca.crt
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.183071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.183113    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.183139    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls podName:d83906ba-ac2d-453a-b42b-3a3d958834ee nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.183124394 +0000 UTC m=+73.321401339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-8vh2s" (UID: "d83906ba-ac2d-453a-b42b-3a3d958834ee") : secret "samples-operator-tls" not found
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.183161    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls podName:9cf495d7-831a-4266-a508-c12d9968971d nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.183150662 +0000 UTC m=+73.321427605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls") pod "image-registry-6f4bdbf4dd-bqs7h" (UID: "9cf495d7-831a-4266-a508-c12d9968971d") : secret "image-registry-tls" not found
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.183184    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls podName:ef6cea18-3a39-4c4b-8fba-0284ca6f2e98 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.183175162 +0000 UTC m=+73.321452108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jnx8b" (UID: "ef6cea18-3a39-4c4b-8fba-0284ca6f2e98") : secret "cluster-monitoring-operator-tls" not found
Nov 03 16:10:30.183303 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.183228    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert podName:d671aa4a-c3a4-474d-8b2c-9b60ae49c0de nodeName:}" failed. No retries permitted until 2026-11-03 16:10:38.183195271 +0000 UTC m=+73.321472214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z4krf" (UID: "d671aa4a-c3a4-474d-8b2c-9b60ae49c0de") : secret "networking-console-plugin-cert" not found
Nov 03 16:10:30.337864 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.337789    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"]
Nov 03 16:10:30.340153 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:30.340126    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447a7d8a_1272_419d_a815_6b4353a2011a.slice/crio-d7041d0dc8b8f37f88844291ea5239eccbcebf86ad9d397d2165ae7370b192b2 WatchSource:0}: Error finding container d7041d0dc8b8f37f88844291ea5239eccbcebf86ad9d397d2165ae7370b192b2: Status 404 returned error can't find the container with id d7041d0dc8b8f37f88844291ea5239eccbcebf86ad9d397d2165ae7370b192b2
Nov 03 16:10:30.371779 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.371754    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"]
Nov 03 16:10:30.374560 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:30.374533    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c0882b_45cb_47a7_983e_01e5748ffab9.slice/crio-1cefbaa9a0d8c78540cfcbc3c9a116f39c1cc20a3d8dbb979fb2341b35bd10cb WatchSource:0}: Error finding container 1cefbaa9a0d8c78540cfcbc3c9a116f39c1cc20a3d8dbb979fb2341b35bd10cb: Status 404 returned error can't find the container with id 1cefbaa9a0d8c78540cfcbc3c9a116f39c1cc20a3d8dbb979fb2341b35bd10cb
Nov 03 16:10:30.385381 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.385357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:30.385475 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.385398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:30.385529 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.385475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:30.385529 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.385478    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:30.385614 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.385544    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:32.3855225 +0000 UTC m=+67.523799456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:30.385614 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.385564    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:30.385614 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.385607    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:32.385597538 +0000 UTC m=+67.523874484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:30.385719 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.385700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:30.433891 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.433858    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:10:30.434021 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.433946    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:10:30.436035 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.436007    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 16:10:30.436102 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:30.436090    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs podName:64a4a725-bb12-428a-a3d0-62622448f577 nodeName:}" failed. No retries permitted until 2026-11-03 16:11:34.436074522 +0000 UTC m=+129.574351474 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs") pod "network-metrics-daemon-qd8hz" (UID: "64a4a725-bb12-428a-a3d0-62622448f577") : secret "metrics-daemon-secret" not found
Nov 03 16:10:30.438135 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.438118    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:30.486662 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.486629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:30.489320 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.489297    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:30.604316 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.604233    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7zlt\""
Nov 03 16:10:30.606274 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.606255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:30.608050 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.608030    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zsc2l\""
Nov 03 16:10:30.610163 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.610138    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:30.642685 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.642652    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p" event={"ID":"b5c0882b-45cb-47a7-983e-01e5748ffab9","Type":"ContainerStarted","Data":"1cefbaa9a0d8c78540cfcbc3c9a116f39c1cc20a3d8dbb979fb2341b35bd10cb"}
Nov 03 16:10:30.644136 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.644111    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh" event={"ID":"447a7d8a-1272-419d-a815-6b4353a2011a","Type":"ContainerStarted","Data":"d7041d0dc8b8f37f88844291ea5239eccbcebf86ad9d397d2165ae7370b192b2"}
Nov 03 16:10:30.646723 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.646700    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv" event={"ID":"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e","Type":"ContainerStarted","Data":"3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32"}
Nov 03 16:10:30.646830 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.646741    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv" event={"ID":"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e","Type":"ContainerStarted","Data":"4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779"}
Nov 03 16:10:30.757925 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.757854    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv" podStartSLOduration=2.757839606 podStartE2EDuration="2.757839606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:30.757065309 +0000 UTC m=+65.895342270" watchObservedRunningTime="2026-11-03 16:10:30.757839606 +0000 UTC m=+65.896116567"
Nov 03 16:10:30.990988 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.990959    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7sv8"]
Nov 03 16:10:30.993714 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:30.993682    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0366db_bfff_4d9c_af6e_37b79c084739.slice/crio-e63f8f3c23e4ef587acc75e0dd1f95048caeb64db9cdb7562623f997e90e70fc WatchSource:0}: Error finding container e63f8f3c23e4ef587acc75e0dd1f95048caeb64db9cdb7562623f997e90e70fc: Status 404 returned error can't find the container with id e63f8f3c23e4ef587acc75e0dd1f95048caeb64db9cdb7562623f997e90e70fc
Nov 03 16:10:30.993714 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:30.993710    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cs98m"]
Nov 03 16:10:31.027087 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:31.027058    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16b1c0a_5c06_4fef_9a5a_664d92a24a25.slice/crio-c12ce374fa0819b63d7942fb7a52b82ab74d5835c1c2af6dc7b638db0e52f11f WatchSource:0}: Error finding container c12ce374fa0819b63d7942fb7a52b82ab74d5835c1c2af6dc7b638db0e52f11f: Status 404 returned error can't find the container with id c12ce374fa0819b63d7942fb7a52b82ab74d5835c1c2af6dc7b638db0e52f11f
Nov 03 16:10:31.121122 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.121086    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7qth8"]
Nov 03 16:10:31.125064 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.125039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.179105 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.179073    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 16:10:31.179283 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.179148    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cjszs\""
Nov 03 16:10:31.179862 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.179838    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7qth8"]
Nov 03 16:10:31.180477 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.180459    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 16:10:31.180529 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.180517    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 16:10:31.193119 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.193091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24746226-0fdb-40fa-a877-e889fdae7daf-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.193119 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.193124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6n52\" (UniqueName: \"kubernetes.io/projected/24746226-0fdb-40fa-a877-e889fdae7daf-kube-api-access-h6n52\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.193315 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.193252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24746226-0fdb-40fa-a877-e889fdae7daf-signing-key\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.279376 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.279296    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 16:10:31.293753 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.293722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24746226-0fdb-40fa-a877-e889fdae7daf-signing-key\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.293940 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.293907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24746226-0fdb-40fa-a877-e889fdae7daf-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.294042 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.293948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6n52\" (UniqueName: \"kubernetes.io/projected/24746226-0fdb-40fa-a877-e889fdae7daf-kube-api-access-h6n52\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.294758 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.294735    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24746226-0fdb-40fa-a877-e889fdae7daf-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.296709 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.296681    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24746226-0fdb-40fa-a877-e889fdae7daf-signing-key\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.350015 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.349975    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6n52\" (UniqueName: \"kubernetes.io/projected/24746226-0fdb-40fa-a877-e889fdae7daf-kube-api-access-h6n52\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.437259 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.437222    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:31.652125 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.652086    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cs98m" event={"ID":"a16b1c0a-5c06-4fef-9a5a-664d92a24a25","Type":"ContainerStarted","Data":"d73ba765f1a0d33734ff9b1f6b018b6e0dd65bd9e938db822794b1548aaa3234"}
Nov 03 16:10:31.652611 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.652148    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cs98m" event={"ID":"a16b1c0a-5c06-4fef-9a5a-664d92a24a25","Type":"ContainerStarted","Data":"c12ce374fa0819b63d7942fb7a52b82ab74d5835c1c2af6dc7b638db0e52f11f"}
Nov 03 16:10:31.652952 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.652926    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:31.654565 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.654534    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7sv8" event={"ID":"2b0366db-bfff-4d9c-af6e-37b79c084739","Type":"ContainerStarted","Data":"e63f8f3c23e4ef587acc75e0dd1f95048caeb64db9cdb7562623f997e90e70fc"}
Nov 03 16:10:31.813655 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.813590    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cs98m" podStartSLOduration=66.813570056 podStartE2EDuration="1m6.813570056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:31.762622308 +0000 UTC m=+66.900899272" watchObservedRunningTime="2026-11-03 16:10:31.813570056 +0000 UTC m=+66.951847017"
Nov 03 16:10:31.816388 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:31.816358    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7qth8"]
Nov 03 16:10:31.835322 ip-10-0-137-122 kubenswrapper[2582]: W1103 16:10:31.835283    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24746226_0fdb_40fa_a877_e889fdae7daf.slice/crio-896b9a49026b80b09913012a14030199df088847332abe860174c5edf0a30123 WatchSource:0}: Error finding container 896b9a49026b80b09913012a14030199df088847332abe860174c5edf0a30123: Status 404 returned error can't find the container with id 896b9a49026b80b09913012a14030199df088847332abe860174c5edf0a30123
Nov 03 16:10:32.406465 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.406423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:32.406636 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.406476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:32.406636 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:32.406549    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:32.406636 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:32.406623    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:36.406604591 +0000 UTC m=+71.544881550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:32.406759 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:32.406658    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:32.406788 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:32.406777    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:36.406762703 +0000 UTC m=+71.545039645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:32.661550 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.661465    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p" event={"ID":"b5c0882b-45cb-47a7-983e-01e5748ffab9","Type":"ContainerStarted","Data":"c71ebf5bfa331023441ee01597abd87ffd84428875ca2d8e5b56ec5570ee9cd6"}
Nov 03 16:10:32.663733 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.663702    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh" event={"ID":"447a7d8a-1272-419d-a815-6b4353a2011a","Type":"ContainerStarted","Data":"13dd714fed67226a7a4050426aa7d927fa3a778adf7da99661292f2157a6e62b"}
Nov 03 16:10:32.667048 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.667017    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8" event={"ID":"24746226-0fdb-40fa-a877-e889fdae7daf","Type":"ContainerStarted","Data":"03e4b1dceae02aaf6f22614b55f4045532a33accbd40a6a37dc5c3ecf55cbf04"}
Nov 03 16:10:32.667183 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.667061    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8" event={"ID":"24746226-0fdb-40fa-a877-e889fdae7daf","Type":"ContainerStarted","Data":"896b9a49026b80b09913012a14030199df088847332abe860174c5edf0a30123"}
Nov 03 16:10:32.670271 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.670246    2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:32.670383 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.670293    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:32.670631 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.670615    2582 scope.go:122] "RemoveContainer" containerID="5baeab839e0d08d18a7338066961f762077bc9655868802f6ab04a19c2afbe98"
Nov 03 16:10:32.670807 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:32.670790    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l2th2_openshift-console-operator(290e7793-2a71-4dfb-9375-be6c7d26d7bb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2" podUID="290e7793-2a71-4dfb-9375-be6c7d26d7bb"
Nov 03 16:10:32.742674 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.742645    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7nqs4_7ff34aa0-6133-41ff-b88a-f5360d6d2f90/dns-node-resolver/0.log"
Nov 03 16:10:32.889668 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.889610    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p" podStartSLOduration=2.198862788 podStartE2EDuration="3.889590115s" totalImagesPullingTime="1.690727327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:32.774404836 +0000 UTC m=+67.912681800" watchObservedRunningTime="2026-11-03 16:10:32.889590115 +0000 UTC m=+68.027867077"
Nov 03 16:10:32.889832 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:32.889724    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8" podStartSLOduration=1.889714003 podStartE2EDuration="1.889714003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:32.887904973 +0000 UTC m=+68.026181937" watchObservedRunningTime="2026-11-03 16:10:32.889714003 +0000 UTC m=+68.027990965"
Nov 03 16:10:33.005967 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:33.005863    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh" podStartSLOduration=2.504808855 podStartE2EDuration="4.00584827s" totalImagesPullingTime="1.501039415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:33.005465624 +0000 UTC m=+68.143742585" watchObservedRunningTime="2026-11-03 16:10:33.00584827 +0000 UTC m=+68.144125232"
Nov 03 16:10:34.142005 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:34.141946    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mxwfz_203b34a5-c159-485a-83fd-34cf7c348b9d/node-ca/0.log"
Nov 03 16:10:34.543836 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:34.543805    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v4tnz_a75bf77f-4e2f-43d3-aa27-1ee258dc4db6/serve-healthcheck-canary/0.log"
Nov 03 16:10:34.741940 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:34.741915    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-2b5rh_447a7d8a-1272-419d-a815-6b4353a2011a/migrator/0.log"
Nov 03 16:10:34.942128 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:34.942098    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-svr6p_b5c0882b-45cb-47a7-983e-01e5748ffab9/migrator/0.log"
Nov 03 16:10:35.145010 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:35.144982    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b_43e7bf8b-0e8b-46e5-a55a-12dd965a41d0/kube-storage-version-migrator-operator/0.log"
Nov 03 16:10:36.449576 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:36.449531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:36.450165 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:36.449583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:36.450165 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:36.449663    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:36.450165 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:36.449742    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:44.449722931 +0000 UTC m=+79.587999891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:36.450165 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:36.449754    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:36.450165 ip-10-0-137-122 kubenswrapper[2582]: E1103 16:10:36.449793    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:44.449782036 +0000 UTC m=+79.588058995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:38.264678 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:38.265170 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:38.265170 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:38.265170 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:38.265170 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:38.265170 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:38.265170 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:38.265170 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.264909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:38.265599 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.265479    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:38.268130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.268104    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:38.268130 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.268122    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:38.268340 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.268241    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:38.268650 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.268629    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:38.268760 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.268740    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:38.269485 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.269465    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:38.269781 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.269762    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:38.329447 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.329391    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cqnft\""
Nov 03 16:10:38.331531 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.331514    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:38.355940 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.355924    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dgbbj\""
Nov 03 16:10:38.358038 ip-10-0-137-122 kubenswrapper[2582]: I1103 16:10:38.358021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:38.360299 ip-10-0-137-122 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 16:10:38.380592 ip-10-0-137-122 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 16:10:38.380768 ip-10-0-137-122 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 16:10:38.380965 ip-10-0-137-122 systemd[1]: kubelet.service: Consumed 4.023s CPU time.
Nov 03 16:10:38.390897 ip-10-0-137-122 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 16:10:38.464733 ip-10-0-137-122 kubenswrapper[5112]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:10:38.464733 ip-10-0-137-122 kubenswrapper[5112]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 16:10:38.464733 ip-10-0-137-122 kubenswrapper[5112]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:10:38.464733 ip-10-0-137-122 kubenswrapper[5112]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 16:10:38.466718 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466643    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.466718 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466716    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.466718 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466720    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466723    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466727    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466730    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466732    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466736    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466739    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466741    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466744    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466746    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466749    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466752    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466754    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466757    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466760    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466762    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466765    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466768    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466770    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466780    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.466849 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466783    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466786    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466789    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466792    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466794    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466797    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466799    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466802    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466804    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466808    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466811    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466813    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466816    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466819    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466822    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466825    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466827    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466831    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466833    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.467496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466836    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466838    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466841    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466845    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466849    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466853    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466857    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466860    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466863    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466866    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466869    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466873    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466876    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466879    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466882    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466885    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466888    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466891    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.467980 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466893    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466896    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466899    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466901    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466904    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466907    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466909    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466912    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466915    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466918    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466920    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466923    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466925    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466934    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466937    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466940    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466943    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466945    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466948    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466951    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.468451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466953    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466956    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466958    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466961    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466964    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466966    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466969    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466972    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466974    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466977    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466980    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466982    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466985    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466988    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466990    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466993    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466996    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.466998    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467001    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467004    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.468953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467007    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467010    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467012    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467015    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467019    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467021    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467024    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467781    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467789    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467793    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467796    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467800    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467805    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467808    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467811    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467814    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467816    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467819    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467822    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.469484 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467824    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467827    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467829    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467832    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467835    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467838    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467841    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467844    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467847    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467849    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467852    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467854    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467857    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467861    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467863    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467866    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467868    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467871    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467873    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467876    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.469962 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467879    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467882    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467885    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467887    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467890    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467892    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467895    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467897    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467900    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467903    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467906    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467909    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467911    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467914    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467917    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467919    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467922    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467925    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467927    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467930    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.470496 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467932    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467935    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467937    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467940    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467942    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467945    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467948    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467950    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467960    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467963    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467966    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467968    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467971    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467974    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467977    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467980    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467982    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467985    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467988    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.471038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467991    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467994    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467997    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.467999    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468002    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468005    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468007    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468012    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468016    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468019    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468022    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468025    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468028    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468031    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468034    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468036    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468039    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468042    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468044    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.471532 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468047    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468049    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468052    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468054    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468057    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468059    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468062    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468065    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468072    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468075    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468078    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468080    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468083    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468085    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468088    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468090    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468599    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468605    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468608    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.472009 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468610    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468613    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468616    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468619    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468621    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468624    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468627    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468629    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468632    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468638    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468641    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468644    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468647    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468649    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468652    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468655    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468658    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468660    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468663    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.472689 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468666    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468668    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468671    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468674    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468677    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468679    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468682    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468684    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468687    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468690    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468692    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468695    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468697    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468700    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468702    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468707    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468711    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468714    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468717    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.473551 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468720    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468723    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468725    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468729    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468732    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468734    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468738    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468740    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468743    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468746    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468748    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468751    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468753    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468756    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468758    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468761    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468764    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468768    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468770    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468773    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.474422 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468776    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468778    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468781    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468784    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468787    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468790    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468793    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468796    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468798    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468800    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468803    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468805    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468808    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468810    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468813    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468817    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468820    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468822    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468825    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.475347 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468828    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468831    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468833    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468836    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468838    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468841    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468843    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468846    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468848    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468851    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468859    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468862    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468864    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468867    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468869    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468872    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468874    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468877    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468879    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.476236 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468882    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468890    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468893    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468895    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468898    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468900    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.468904    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469005    5112 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469012    5112 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469019    5112 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469024    5112 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469028    5112 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469031    5112 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469036    5112 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469041    5112 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469044    5112 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469047    5112 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469051    5112 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469054    5112 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469057    5112 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469060    5112 flags.go:64] FLAG: --cgroup-root=""
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469063    5112 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469066    5112 flags.go:64] FLAG: --client-ca-file=""
Nov 03 16:10:38.476975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469069    5112 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469078    5112 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469085    5112 flags.go:64] FLAG: --cluster-domain=""
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469088    5112 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469091    5112 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469094    5112 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469098    5112 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469102    5112 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469105    5112 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469108    5112 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469112    5112 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469115    5112 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469118    5112 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469136    5112 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469141    5112 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469144    5112 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469149    5112 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469152    5112 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469157    5112 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469160    5112 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469163    5112 flags.go:64] FLAG: --enable-server="true"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469167    5112 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469175    5112 flags.go:64] FLAG: --event-burst="100"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469179    5112 flags.go:64] FLAG: --event-qps="50"
Nov 03 16:10:38.477826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469182    5112 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469185    5112 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469188    5112 flags.go:64] FLAG: --eviction-hard=""
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469192    5112 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469209    5112 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469212    5112 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469215    5112 flags.go:64] FLAG: --eviction-soft=""
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469219    5112 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469222    5112 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469224    5112 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469233    5112 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469237    5112 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469240    5112 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469243    5112 flags.go:64] FLAG: --feature-gates=""
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469247    5112 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469250    5112 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469253    5112 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469256    5112 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469260    5112 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469263    5112 flags.go:64] FLAG: --help="false"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469266    5112 flags.go:64] FLAG: --hostname-override="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469269    5112 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469272    5112 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 16:10:38.478475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469275    5112 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469279    5112 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469282    5112 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469286    5112 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469290    5112 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469292    5112 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469297    5112 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469300    5112 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469303    5112 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469306    5112 flags.go:64] FLAG: --kube-reserved=""
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469309    5112 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469312    5112 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469315    5112 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469318    5112 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469320    5112 flags.go:64] FLAG: --lock-file=""
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469323    5112 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469326    5112 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469329    5112 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469335    5112 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469338    5112 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469346    5112 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469349    5112 flags.go:64] FLAG: --logging-format="text"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469352    5112 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469356    5112 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 16:10:38.479046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469359    5112 flags.go:64] FLAG: --manifest-url=""
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469362    5112 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469366    5112 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469369    5112 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469373    5112 flags.go:64] FLAG: --max-pods="110"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469376    5112 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469379    5112 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469382    5112 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469385    5112 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469388    5112 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469391    5112 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469395    5112 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469406    5112 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469409    5112 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469413    5112 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469416    5112 flags.go:64] FLAG: --pod-cidr=""
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469419    5112 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469422    5112 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469425    5112 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469428    5112 flags.go:64] FLAG: --port="10250"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469431    5112 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469434    5112 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00053910f14216d9e"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469437    5112 flags.go:64] FLAG: --qos-reserved=""
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469440    5112 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469443    5112 flags.go:64] FLAG: --register-node="true"
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469446    5112 flags.go:64] FLAG: --register-with-taints=""
Nov 03 16:10:38.479782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469450    5112 flags.go:64] FLAG: --registry-burst="10"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469452    5112 flags.go:64] FLAG: --registry-qps="5"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469455    5112 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469464    5112 flags.go:64] FLAG: --reserved-memory=""
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469468    5112 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469471    5112 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469474    5112 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469477    5112 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469480    5112 flags.go:64] FLAG: --runonce="false"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469483    5112 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469486    5112 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469489    5112 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469492    5112 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469495    5112 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469498    5112 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469501    5112 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469504    5112 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469509    5112 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469512    5112 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469514    5112 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469522    5112 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469525    5112 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469528    5112 flags.go:64] FLAG: --system-cgroups=""
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469531    5112 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469543    5112 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 16:10:38.480509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469546    5112 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469549    5112 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469556    5112 flags.go:64] FLAG: --tls-min-version=""
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469558    5112 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469561    5112 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469564    5112 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469567    5112 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469570    5112 flags.go:64] FLAG: --v="2"
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469574    5112 flags.go:64] FLAG: --version="false"
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469578    5112 flags.go:64] FLAG: --vmodule=""
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469582    5112 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.469595    5112 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469770    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469774    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469777    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469781    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469784    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469787    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469790    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469793    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469795    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.481325 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469798    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469801    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469803    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469807    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469810    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469812    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469816    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469819    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469821    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469824    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469827    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469830    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469832    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469835    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469837    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469840    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469843    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469845    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469848    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469852    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.481953 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469855    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469858    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469861    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469869    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469873    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469875    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469878    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469880    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469883    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469887    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469890    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469893    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469895    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469898    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469901    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469904    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469907    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469910    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469913    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.482561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469916    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469918    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469921    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469924    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469926    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469929    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469932    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469934    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469937    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469939    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469942    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469945    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469947    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469950    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469952    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469955    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469957    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469965    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469969    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469972    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.483038 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469974    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469977    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469979    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469982    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469985    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469987    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469990    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469995    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.469999    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470002    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470004    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470008    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470010    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470013    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470016    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470018    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470021    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470023    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470026    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470028    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.483608 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470031    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470033    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470036    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470038    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470041    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470044    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470046    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470048    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470051    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470053    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470057    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470059    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470062    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470064    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470067    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470069    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470071    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.484280 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470074    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.470100    5112 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470139    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470143    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470146    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470149    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470153    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470156    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470159    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470162    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470164    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470167    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470170    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470172    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470175    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.484818 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470177    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470180    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470183    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470186    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470188    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470191    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470193    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470208    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470211    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470213    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470216    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470220    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470223    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470226    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470229    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470231    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470234    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470237    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470240    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470242    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.485192 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470245    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470249    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470252    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470258    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470261    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470264    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470267    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470270    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470272    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470275    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470277    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470280    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470283    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470285    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470288    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470290    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470293    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470296    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470299    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.485787 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470301    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470304    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470306    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470309    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470311    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470315    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470317    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470320    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470323    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470325    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470328    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470330    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470333    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470336    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470338    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470341    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470345    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470347    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470350    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470353    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.486273 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470355    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470358    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470360    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470363    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470365    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470367    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470370    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470373    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470375    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470378    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470380    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470383    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470385    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470388    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470390    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470393    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470396    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470399    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470403    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470405    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.486777 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470408    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470410    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470413    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470416    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470418    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470421    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470423    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470426    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470429    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470432    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470435    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470437    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470440    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.470443    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.475385    5112 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.475404    5112 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.475425    5112 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.475432    5112 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475540    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.487331 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475548    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475553    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475558    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475562    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475567    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475572    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475576    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475581    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475585    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475589    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475593    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475597    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475603    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475607    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475611    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475616    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475620    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475624    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475628    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475632    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.487801 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475637    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475641    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475648    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475655    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475660    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475665    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475669    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475673    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475685    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475690    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475695    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475699    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475703    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475707    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475712    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475716    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475720    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475724    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475729    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.488429 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475733    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475745    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475750    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475754    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475758    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475762    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475767    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475773    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475780    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475784    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475790    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475794    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475799    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475803    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475807    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475811    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475815    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475820    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475824    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475828    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.488895 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475832    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475837    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475849    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475853    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475857    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475861    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475866    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475870    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475875    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475879    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475884    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475888    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475892    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475897    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475901    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475905    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475909    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475914    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475918    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475923    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.489451 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475927    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475931    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475935    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475940    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475944    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475948    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475952    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475957    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475961    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475966    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475971    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475975    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475979    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475984    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475988    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.475999    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476004    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476008    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476012    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.490003 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476017    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476021    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476025    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476029    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476033    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476038    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476042    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.476083    5112 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476745    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476874    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476880    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476885    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476891    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476896    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476902    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.490598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476906    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476911    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476915    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476926    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476930    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476936    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476941    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476946    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476950    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476955    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476959    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476964    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476969    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.476974    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477015    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477022    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477033    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477037    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477042    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477046    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.491149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477051    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477055    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477060    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477064    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477069    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477074    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477078    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477083    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477092    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477097    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477101    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477105    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477110    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477114    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477119    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477123    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477128    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477132    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477140    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477145    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.491892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477149    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477159    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477164    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477169    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477173    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477178    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477182    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477234    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477241    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477246    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477250    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477254    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477259    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477264    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477272    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477278    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477282    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477286    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477291    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477296    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.492615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477301    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477306    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477310    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477315    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477320    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477324    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477329    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477343    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477350    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477355    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477364    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477369    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477373    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477433    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477459    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477464    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477469    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477475    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477480    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.493388 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477485    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477490    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477494    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477499    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477503    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477507    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477512    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477522    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477526    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477531    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477540    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477548    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477553    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477558    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477562    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477565    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477588    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477617    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477716    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.494149 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477721    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.477756    5112 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477811    5112 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477816    5112 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477820    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477823    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477825    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477828    5112 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477831    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477834    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477837    5112 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477842    5112 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477844    5112 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477847    5112 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477851    5112 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477854    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 16:10:38.494749 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477857    5112 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477860    5112 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477862    5112 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477865    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477867    5112 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477870    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477873    5112 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477876    5112 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477880    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477883    5112 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477887    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477890    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477893    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477895    5112 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477898    5112 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477901    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477904    5112 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477907    5112 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477910    5112 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477912    5112 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 16:10:38.495141 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477915    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477918    5112 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477920    5112 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477923    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477925    5112 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477928    5112 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477930    5112 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477933    5112 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477935    5112 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477938    5112 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477940    5112 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477943    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477945    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477948    5112 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477951    5112 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477953    5112 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477956    5112 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477958    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477962    5112 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477965    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 16:10:38.495747 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477967    5112 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477970    5112 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477972    5112 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477974    5112 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477977    5112 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477980    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477982    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477985    5112 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477989    5112 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477991    5112 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477994    5112 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477996    5112 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.477999    5112 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478002    5112 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478004    5112 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478007    5112 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478010    5112 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478012    5112 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478014    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 16:10:38.496301 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478017    5112 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478022    5112 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478027    5112 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478030    5112 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478032    5112 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478035    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478038    5112 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478040    5112 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478043    5112 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478046    5112 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478048    5112 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478051    5112 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478053    5112 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478056    5112 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478058    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478061    5112 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478063    5112 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478066    5112 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478068    5112 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 16:10:38.496838 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478072    5112 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478074    5112 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478077    5112 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478080    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478085    5112 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478089    5112 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478092    5112 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478095    5112 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478097    5112 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478100    5112 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478102    5112 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478105    5112 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478107    5112 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:38.478110    5112 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.478161    5112 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.479083    5112 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.479132    5112 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.479321    5112 server.go:1014] "Starting client certificate rotation"
Nov 03 16:10:38.497378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.479483    5112 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.479548    5112 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:04:25 +0000 UTC" deadline="2028-07-04 02:42:04.289577106 +0000 UTC"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.479595    5112 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14602h31m25.809985428s"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.479761    5112 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.480782    5112 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.481940    5112 log.go:25] "Validated CRI v1 runtime API"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.483562    5112 log.go:25] "Validated CRI v1 image API"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.484288    5112 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 16:10:38.497978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.486221    5112 fs.go:135] Filesystem UUIDs: map[74ad188d-f11d-473e-97a8-bdfea535610e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e396b55a-86e3-4eff-ac99-4a9ddd441543:/dev/nvme0n1p4]
Nov 03 16:10:38.498406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.486241    5112 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0733f0ad5e5ae264f06ec8aa03f457f50b0123458d2580d34fd80209d4d6a2ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0733f0ad5e5ae264f06ec8aa03f457f50b0123458d2580d34fd80209d4d6a2ae/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cefbaa9a0d8c78540cfcbc3c9a116f39c1cc20a3d8dbb979fb2341b35bd10cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cefbaa9a0d8c78540cfcbc3c9a116f39c1cc20a3d8dbb979fb2341b35bd10cb/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20d25fae91f0b396d21563561efdfd1b0e72e5696f63ad195d5f21918944a802/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20d25fae91f0b396d21563561efdfd1b0e72e5696f63ad195d5f21918944a802/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/231d0cf57d8714e43577c44dbdb66bd96cf2040e27b71c48ae813c5166f4d77b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/231d0cf57d8714e43577c44dbdb66bd96cf2040e27b71c48ae813c5166f4d77b/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30468e0b0f5f72ffe3f2e9a8513b606f743f8bab828a3e2eceac5555d535d23a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30468e0b0f5f72ffe3f2e9a8513b606f743f8bab828a3e2eceac5555d535d23a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d34a9f85672264c2ee4b2649d9e8db0810adbff1a134594fb3f21b1dbf9c524/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d34a9f85672264c2ee4b2649d9e8db0810adbff1a134594fb3f21b1dbf9c524/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/508e37556cb72662a36c5f780d4798bd85b0d31d83c83e7fd23212f11a3e0914/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/508e37556cb72662a36c5f780d4798bd85b0d31d83c83e7fd23212f11a3e0914/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61f1def4221d45f9b717029ef082e79ec61eb29161d715aad69795a18a3f4b70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61f1def4221d45f9b717029ef082e79ec61eb29161d715aad69795a18a3f4b70/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c71a4b665035f72770ba76a9ac8a6d4e348e8a19a6437e89fc09ce5135725fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c71a4b665035f72770ba76a9ac8a6d4e348e8a19a6437e89fc09ce5135725fb/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71467ccfa2ab2d54da1b774f3abb5223d2a7140599b31ef85597b28a347b3108/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71467ccfa2ab2d54da1b774f3abb5223d2a7140599b31ef85597b28a347b3108/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dd70c2f054731ac3591729fa400fe23995450439706c52b87b7d22e4933c329/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dd70c2f054731ac3591729fa400fe23995450439706c52b87b7d22e4933c329/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/814f0b4743c0ec6d09533d85123858f6e90b9e9ee04bd96edb8e3544da3e60e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/814f0b4743c0ec6d09533d85123858f6e90b9e9ee04bd96edb8e3544da3e60e3/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/864b592a1151b62cfee3a739c8a2383ff72aa8031ed2643e8c9501dbb6727a89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/864b592a1151b62cfee3a739c8a2383ff72aa8031ed2643e8c9501dbb6727a89/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8666150bf8975e316066a5130ded1b551fb1808b983e7906a4c97f3b1f54fa25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8666150bf8975e316066a5130ded1b551fb1808b983e7906a4c97f3b1f54fa25/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/896b9a49026b80b09913012a14030199df088847332abe860174c5edf0a30123/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/896b9a49026b80b09913012a14030199df088847332abe860174c5edf0a30123/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cfdaeb4197a9a33728c70254a2094bad01b8bec4ea357f4c752fe1061fa0914/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cfdaeb4197a9a33728c70254a2094bad01b8bec4ea357f4c752fe1061fa0914/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a11b624f51216074852b68e061ed288d3ee79d5564a9ef1cf3ef9c03713287a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a11b624f51216074852b68e061ed288d3ee79d5564a9ef1cf3ef9c03713287a9/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c12ce374fa0819b63d7942fb7a52b82ab74d5835c1c2af6dc7b638db0e52f11f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c12ce374fa0819b63d7942fb7a52b82ab74d5835c1c2af6dc7b638db0e52f11f/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7041d0dc8b8f37f88844291ea5239eccbcebf86ad9d397d2165ae7370b192b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7041d0dc8b8f37f88844291ea5239eccbcebf86ad9d397d2165ae7370b192b2/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc9350bb5c534b03b6eb51184331bf12244e699ad8f4cf55b5f0bbd32d90cef2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc9350bb5c534b03b6eb51184331bf12244e699ad8f4cf55b5f0bbd32d90cef2/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de8eec11ad8ce3db1e65028d91b707b2571f2a0ea697158f4d0eccafa5f2dc34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de8eec11ad8ce3db1e65028d91b707b2571f2a0ea697158f4d0eccafa5f2dc34/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e63f8f3c23e4ef587acc75e0dd1f95048caeb64db9cdb7562623f997e90e70fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e63f8f3c23e4ef587acc75e0dd1f95048caeb64db9cdb7562623f997e90e70fc/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1ef976c87d917640602d55433763af947237fb0c2c8f4f834f01a101a3858b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1ef976c87d917640602d55433763af947237fb0c2c8f4f834f01a101a3858b1/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f/volumes/kubernetes.io~projected/kube-api-access-cxrmz:{mountpoint:/var/lib/kubelet/pods/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f/volumes/kubernetes.io~projected/kube-api-access-cxrmz major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc284dd-fff4-439d-8732-7b78daf69085/volumes/kubernetes.io~projected/kube-api-access-hwrz8:{mountpoint:/var/lib/kubelet/pods/0dc284dd-fff4-439d-8732-7b78daf69085/volumes/kubernetes.io~projected/kube-api-access-hwrz8 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11ba431e-e81b-4dbf-9d65-445a66ff6e07/volumes/kubernetes.io~projected/kube-api-access-4w872:{mountpoint:/var/lib/kubelet/pods/11ba431e-e81b-4dbf-9d65-445a66ff6e07/volumes/kubernetes.io~projected/kube-api-access-4w872 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/203b34a5-c159-485a-83fd-34cf7c348b9d/volumes/kubernetes.io~projected/kube-api-access-57sfl:{mountpoint:/var/lib/kubelet/pods/203b34a5-c159-485a-83fd-34cf7c348b9d/volumes/kubernetes.io~projected/kube-api-access-57sfl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24746226-0fdb-40fa-a877-e889fdae7daf/volumes/kubernetes.io~projected/kube-api-access-h6n52:{mountpoint:/var/lib/kubelet/pods/24746226-0fdb-40fa-a877-e889fdae7daf/volumes/kubernetes.io~projected/kube-api-access-h6n52 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24746226-0fdb-40fa-a877-e889fdae7daf/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/24746226-0fdb-40fa-a877-e889fdae7daf/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290e7793-2a71-4dfb-9375-be6c7d26d7bb/volumes/kubernetes.io~projected/kube-api-access-gtsj4:{mountpoint:/var/lib/kubelet/pods/290e7793-2a71-4dfb-9375-be6c7d26d7bb/volumes/kubernetes.io~projected/kube-api-access-gtsj4 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290e7793-2a71-4dfb-9375-be6c7d26d7bb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/290e7793-2a71-4dfb-9375-be6c7d26d7bb/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b0366db-bfff-4d9c-af6e-37b79c084739/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2b0366db-bfff-4d9c-af6e-37b79c084739/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31c078e5-03f1-4b2e-ba52-5c10d58a5d82/volumes/kubernetes.io~projected/kube-api-access-6khfm:{mountpoint:/var/lib/kubelet/pods/31c078e5-03f1-4b2e-ba52-5c10d58a5d82/volumes/kubernetes.io~projected/kube-api-access-6khfm major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~projected/kube-api-access-q4vrz:{mountpoint:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~projected/kube-api-access-q4vrz major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/metrics-certs major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0/volumes/kubernetes.io~projected/kube-api-access-k7jzc:{mountpoint:/var/lib/kubelet/pods/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0/volumes/kubernetes.io~projected/kube-api-access-k7jzc major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43f18aa2-9718-4d99-9e5c-0da5f272cd02/volumes/kubernetes.io~projected/kube-api-access-frnzm:{mountpoint:/var/lib/kubelet/pods/43f18aa2-9718-4d99-9e5c-0da5f272cd02/volumes/kubernetes.io~projected/kube-api-access-frnzm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/447a7d8a-1272-419d-a815-6b4353a2011a/volumes/kubernetes.io~projected/kube-api-access-tp5xr:{mountpoint:/var/lib/kubelet/pods/447a7d8a-1272-419d-a815-6b4353a2011a/volumes/kubernetes.io~projected/kube-api-access-tp5xr major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47333977-43b6-47af-ab13-b5dbee646be5/volumes/kubernetes.io~projected/kube-api-access-79s9t:{mountpoint:/var/lib/kubelet/pods/47333977-43b6-47af-ab13-b5dbee646be5/volumes/kubernetes.io~projected/kube-api-access-79s9t major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e/volumes/kubernetes.io~projected/kube-api-access-tsjk5:{mountpoint:/var/lib/kubelet/pods/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e/volumes/kubernetes.io~projected/kube-api-access-tsjk5 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64a4a725-bb12-428a-a3d0-62622448f577/volumes/kubernetes.io~projected/kube-api-access-hh9g7:{mountpoint:/var/lib/kubelet/pods/64a4a725-bb12-428a-a3d0-62622448f577/volumes/kubernetes.io~projected/kube-api-access-hh9g7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volumes/kubernetes.io~projected/kube-api-access-6qpk4:{mountpoint:/var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volumes/kubernetes.io~projected/kube-api-access-6qpk4 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7351e46a-1bd0-430b-aec5-1f3111e80370/volumes/kubernetes.io~projected/kube-api-access-xv75d:{mountpoint:/var/lib/kubelet/pods/7351e46a-1bd0-430b-aec5-1f3111e80370/volumes/kubernetes.io~projected/kube-api-access-xv75d major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7351e46a-1bd0-430b-aec5-1f3111e80370/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7351e46a-1bd0-430b-aec5-1f3111e80370/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/754fed8d-9c7a-4cee-8806-794ed3330bc0/volumes/kubernetes.io~projected/kube-api-access-7tv8f:{mountpoint:/var/lib/kubelet/pods/754fed8d-9c7a-4cee-8806-794ed3330bc0/volumes/kubernetes.io~projected/kube-api-access-7tv8f major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/754fed8d-9c7a-4cee-8806-794ed3330bc0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/754fed8d-9c7a-4cee-8806-794ed3330bc0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ff34aa0-6133-41ff-b88a-f5360d6d2f90/volumes/kubernetes.io~projected/kube-api-access-l5p8l:{mountpoint:/var/lib/kubelet/pods/7ff34aa0-6133-41ff-b88a-f5360d6d2f90/volumes/kubernetes.io~projected/kube-api-access-l5p8l major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/kube-api-access-lrvcd:{mountpoint:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/kube-api-access-lrvcd major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/registry-tls major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/kube-api-access-jsqfl:{mountpoint:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/kube-api-access-jsqfl major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/registry-tls major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a16b1c0a-5c06-4fef-9a5a-664d92a24a25/volumes/kubernetes.io~projected/kube-api-access-zpljr:{mountpoint:/var/lib/kubelet/pods/a16b1c0a-5c06-4fef-9a5a-664d92a24a25/volumes/kubernetes.io~projected/kube-api-access-zpljr major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6/volumes/kubernetes.io~projected/kube-api-access-mht6z:{mountpoint:/var/lib/kubelet/pods/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6/volumes/kubernetes.io~projected/kube-api-access-mht6z major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6/volumes/kubernetes.io~secret/cert major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1a241b6-335b-4ee8-add5-07a5e71666e5/volumes/kubernetes.io~projected/kube-api-access-h6bwg:{mountpoint:/var/lib/kubelet/pods/b1a241b6-335b-4ee8-add5-07a5e71666e5/volumes/kubernetes.io~projected/kube-api-access-h6bwg major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~projected/kube-api-access-dfwhh:{mountpoint:/var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~projected/kube-api-access-dfwhh major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5c0882b-45cb-47a7-983e-01e5748ffab9/volumes/kubernetes.io~projected/kube-api-access-zg4hz:{mountpoint:/var/lib/kubelet/pods/b5c0882b-45cb-47a7-983e-01e5748ffab9/volumes/kubernetes.io~projected/kube-api-access-zg4hz major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be48d695-434e-473c-a500-a1e2e8c5de24/volumes/kubernetes.io~projected/kube-api-access-wmjrh:{mountpoint:/var/lib/kubelet/pods/be48d695-434e-473c-a500-a1e2e8c5de24/volumes/kubernetes.io~projected/kube-api-access-wmjrh major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c47cd951-b547-4a56-b2c2-1f28b1358ed6/volumes/kubernetes.io~projected/kube-api-access-rvcbz:{mountpoint:/var/lib/kubelet/pods/c47cd951-b547-4a56-b2c2-1f28b1358ed6/volumes/kubernetes.io~projected/kube-api-access-rvcbz major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d79060-9c22-412b-b6df-576c7a2dd340/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d5d79060-9c22-412b-b6df-576c7a2dd340/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83906ba-ac2d-453a-b42b-3a3d958834ee/volumes/kubernetes.io~projected/kube-api-access-w4lrh:{mountpoint:/var/lib/kubelet/pods/d83906ba-ac2d-453a-b42b-3a3d958834ee/volumes/kubernetes.io~projected/kube-api-access-w4lrh major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83906ba-ac2d-453a-b42b-3a3d958834ee/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/d83906ba-ac2d-453a-b42b-3a3d958834ee/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98/volumes/kubernetes.io~projected/kube-api-access-mtrrq:{mountpoint:/var/lib/kubelet/pods/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98/volumes/kubernetes.io~projected/kube-api-access-mtrrq major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7cd889c-edce-4208-9b4f-005d200d566b/volumes/kubernetes.io~projected/kube-api-access-pv7t6:{mountpoint:/var/lib/kubelet/pods/f7cd889c-edce-4208-9b4f-005d200d566b/volumes/kubernetes.io~projected/kube-api-access-pv7t6 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7cd889c-edce-4208-9b4f-005d200d566b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f7cd889c-edce-4208-9b4f-005d200d566b/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff/volumes/kubernetes.io~projected/kube-api-access-nttb7:{mountpoint:/var/lib/kubelet/pods/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff/volumes/kubernetes.io~projected/kube-api-access-nttb7 major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/dc5535eff55f81b3167ba7da0b19ec65880f9ec831defb4159dd9d0891633719/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3654867b7e7cb5382eeca4bdba2bbf130aa3d80ce64724f59b72abd76aa6b732/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7d113dc763ff860240e6b4facb3abb0d40b1fa6637f12f31ac8624ad7271b55e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/37c6e8311ff5a334d365118744b359a963879d71911caca1cd702439139aa7b3/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/16a62ce197bd063db440e5567a322ed4bc5b9a4182be61fc10a5d5696cda7f66/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/173e2e37554c71b47e3bc771100143f3adf427ecfc83b00960a4780e5f833742/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bd649d703a08e6b5ad825d484cc51de46470ed477df1a0f82c1658d6bb8cd980/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c9a5323ac2c7db6090522b835b057b336cdb895ea64fe76d90aba76ac6deb816/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/c38b309cedc463b37c4aae35fd569244c46e3e411d6dcae4655776028db403ae/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/6955c8f9a4382023adccf3e417a047ef71db52ee47584d21cb3a6d22b802fd5d/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/0318854edbb0e066e3a774c446151c3209dc2525540368cc7a3ac8c70cb98219/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/9d421d35bae2f1980334f229cb8b47d57135ec9a7706a536db15fa2842892af9/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1ffc7e1266e80f59e046a608470aa7c1f7af362b384b3f460e91ee85f723fb7f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/811e1dac3cfdf6132d5b0950986a6a429d0f999b4de753b24fe283bcb9bea4d5/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/b78bce17cee2509d2337e7591bcf7048dbb060d037dbc63b293abdacd66b3bb0/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/8b586e237300529d201a7742160338b4e348c8ed8e1a1713731ad10d8b21d8e6/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/e5465529c657848d17460859e68946587f0f8497eb1e60fc31f51f6e3776588a/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/aa0b3414cd818eb5de78d57ab8875899779e2a3ac133b6786591824797b3c543/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/7c36a474f311106d4384a1a044089f020598cfde0af17b75befd6e6602dd01ae/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/5de114f014c4992e3dc3b1ef39fd619ac3596042de5042b55ce8db567182f682/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/753956e7cba02478d1b8f9fc8a73f15ed84b827249e5fda39594d637595a91c4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/d093ca524dcbea8ba7fe59c69f12edd1256129b71b1e24b5df1f79b6ece8d1d0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d045dc24bc866b5c771d3bc2887666d9c0214dca709f2cdb06313487c42bf640/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/7e9dbc23be2c2e2551230b0efb515fdd9b05e0bfd5017bd90f8f7a7be54e3115/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/551a02be272b5b3fba62b4f3ce304c99b3059f59b577ed876ef6d18404aa7141/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1d1b3fd7c0f1f1b70db2bfa485eb1772057c00447fa0e3d29297f9d8ef0ee11e/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/824016c935d6dd41332a8d8e43011736e665958cb159eb8830434fcee50d1f34/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d16b26bcd1934d742d336fa0a3660fffc4323ae4e2fa20a73003fff2cf1c8bc6/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/7a04c2a5f7ede35365b880865fcf40ff9e0b4666002a878660ef623f4eb67a69/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/eaab1e1b581c55a9661ecd480d1a69bb17258f2d5e5c5eb5c0503a245e4b490c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/35badf701c0b12fa3186d32d50aeae843c5db52b5e4aa80f8c1a4ee4362b75e2/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/b1640022cd07682132ca290ff2a6bed35c807f353acb5a00cc0e5d763b9f9d61/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/926b8ed471f2353b64c8348ea83b06c4ede2adbe52b209f6a352203050d1e645/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/166f64552d871451eb27526ef7e751396bd6dc711b1d1cf53d01193c9c6c5718/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/288c8ab43d7044430ea55d851915225f786168167f0ab6bc521e3ef0f28c3375/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/1f6740e0784e09f3fb26b45f0b2f479d67d0c8d4f6bbf543b8e698e0f932f047/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/3f775ebf9da33b9245397e6b89636b0955b0f7ec5e9e0a6a2378fe2c68e51cad/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/1ce9bb43934a580a1669412c7e0694f457585e77f166ee8f696187a62042bcaa/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/d87bb8b03f01e54fcfcb3b1c7258c6f4be49fe39508037e99970291631eb8279/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/34bef01d0142be73d46f9f441bdae8fe89f2543e89e3d6734c86d33d7f7688bf/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/d8227437e233a5a991a301e104c1f735ded2d76d6d987484518a20956ee5e1d6/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/91316a3691f9f6555dcb9f7e18370f59d016023b816c787b61b42fc70764939f/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/46169ffda1c03d58272770d76fd5d1ea0aaac2506c49d7aeb821fabb38a2c525/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/d00d799f3f3e9b83d92d5c22a6b74af08cfea188d1b60ecd8e8ba8419107691d/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/6443d321b3da71b53b5f29b92b1897ef17b34bde908eb034995d02258eae712c/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/345990db844408ff3dc14917fd443cc6f4e4be325a49da07ebd5a7ead03e9cb6/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/c13cd26b0418a596ef91aea0fde9b61e9d6871b2dac57dd993a724799b80ac1e/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/a67a4e146834c4bd7412874f29c039792976499a3887b361c54ec5f381d0eae2/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/99fed651d139f3678eb0063b676315213ab0e713cccceea9575edd6bc47d6fd7/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/7ae457ce1fa208278fc67b85e07222c0502da059999438560244396eeae44178/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/c47893ef6f6073e1387d9cfcd70ba167433ef745c4a203d6cccd5b6b83315c76/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/5aea42e116952c9ec1a2ef092a576e7c73dd6598d9bf044c93dac1b7eadf8c33/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/ad53b6f8f6fd820f87f1b5d104ed3e3d0b4a579eb2d120285555a6cfb368f132/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9b7f6d9812e7e71a745689d6e6b5e3831acaa2d823b7fc788d7cbc8b93609204/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/60c39cda74c3277041a31cfde4733a71b4cdf174bca2340b3ead19b244f05682/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/595d1899ee950c6bafc89c42f346829096da872f4ebaf210f8f96646f61f2959/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/98ee5b4b0c7025467cced21448ecd6ba22afbaf3a55327a0662f822e1f7a13da/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/55ee3136986714a7dabea5d544c8e2e87feb76e3b57d65512a0af4d1859deb67/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/29cc1c812919090fa8dde1effb58d277bf387e8207aedf52ed87e0581ad6617e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5cdd94956292eaa73752510f46327416717bfc261de515e8acbc903f7ad11ce2/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a926dc8e98475d59461cfe343a520b3f3bf928da7d349aee84f5c58fb7bee368/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/dc14a17b8f0cf0abee90ecf3319653f4c498ec696710fd871fc62433b7c0581b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.495360    5112 manager.go:217] Machine: {Timestamp:2026-11-03 16:10:38.494865435 +0000 UTC m=+0.074169280 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101229 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287d1a8f3850709e49007231a82bad SystemUUID:ec287d1a-8f38-5070-9e49-007231a82bad BootID:15fe401b-c2b0-4489-9688-bf3f36df29f1 Filesystems:[{Device:/var/lib/kubelet/pods/754fed8d-9c7a-4cee-8806-794ed3330bc0/volumes/kubernetes.io~projected/kube-api-access-7tv8f DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98/volumes/kubernetes.io~projected/kube-api-access-mtrrq DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61f1def4221d45f9b717029ef082e79ec61eb29161d715aad69795a18a3f4b70/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:200 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~projected/kube-api-access-q4vrz DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/kube-api-access-jsqfl DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cefbaa9a0d8c78540cfcbc3c9a116f39c1cc20a3d8dbb979fb2341b35bd10cb/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24746226-0fdb-40fa-a877-e889fdae7daf/volumes/kubernetes.io~projected/kube-api-access-h6n52 DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/864b592a1151b62cfee3a739c8a2383ff72aa8031ed2643e8c9501dbb6727a89/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e/volumes/kubernetes.io~projected/kube-api-access-tsjk5 DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11ba431e-e81b-4dbf-9d65-445a66ff6e07/volumes/kubernetes.io~projected/kube-api-access-4w872 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8666150bf8975e316066a5130ded1b551fb1808b983e7906a4c97f3b1f54fa25/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a16b1c0a-5c06-4fef-9a5a-664d92a24a25/volumes/kubernetes.io~projected/kube-api-access-zpljr DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc9350bb5c534b03b6eb51184331bf12244e699ad8f4cf55b5f0bbd32d90cef2/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de8eec11ad8ce3db1e65028d91b707b2571f2a0ea697158f4d0eccafa5f2dc34/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:395 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dd70c2f054731ac3591729fa400fe23995450439706c52b87b7d22e4933c329/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:391 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/64a4a725-bb12-428a-a3d0-62622448f577/volumes/kubernetes.io~projected/kube-api-access-hh9g7 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1a241b6-335b-4ee8-add5-07a5e71666e5/volumes/kubernetes.io~projected/kube-api-access-h6bwg DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43f18aa2-9718-4d99-9e5c-0da5f272cd02/volumes/kubernetes.io~projected/kube-api-access-frnzm DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5c0882b-45cb-47a7-983e-01e5748ffab9/volumes/kubernetes.io~projected/kube-api-access-zg4hz DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/290e7793-2a71-4dfb-9375-be6c7d26d7bb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31c078e5-03f1-4b2e-ba52-5c10d58a5d82/volumes/kubernetes.io~projected/kube-api-access-6khfm DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:393 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c47cd951-b547-4a56-b2c2-1f28b1358ed6/volumes/kubernetes.io~projected/kube-api-access-rvcbz DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47333977-43b6-47af-ab13-b5dbee646be5/volumes/kubernetes.io~projected/kube-api-access-79s9t DeviceMajor:0 DeviceMinor:196 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7cd889c-edce-4208-9b4f-005d200d566b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:386 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d79060-9c22-412b-b6df-576c7a2dd340/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volumes/kubernetes.io~projected/kube-api-access-6qpk4 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71467ccfa2ab2d54da1b774f3abb5223d2a7140599b31ef85597b28a347b3108/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c71a4b665035f72770ba76a9ac8a6d4e348e8a19a6437e89fc09ce5135725fb/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30468e0b0f5f72ffe3f2e9a8513b606f743f8bab828a3e2eceac5555d535d23a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0/volumes/kubernetes.io~projected/kube-api-access-k7jzc DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b0366db-bfff-4d9c-af6e-37b79c084739/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be48d695-434e-473c-a500-a1e2e8c5de24/volumes/kubernetes.io~projected/kube-api-access-wmjrh DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7041d0dc8b8f37f88844291ea5239eccbcebf86ad9d397d2165ae7370b192b2/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/203b34a5-c159-485a-83fd-34cf7c348b9d/volumes/kubernetes.io~projected/kube-api-access-57sfl DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d83906ba-ac2d-453a-b42b-3a3d958834ee/volumes/kubernetes.io~projected/kube-api-access-w4lrh DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ff34aa0-6133-41ff-b88a-f5360d6d2f90/volumes/kubernetes.io~projected/kube-api-access-l5p8l DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/814f0b4743c0ec6d09533d85123858f6e90b9e9ee04bd96edb8e3544da3e60e3/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/754fed8d-9c7a-4cee-8806-794ed3330bc0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:392 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff/volumes/kubernetes.io~projected/kube-api-access-nttb7 DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7cd889c-edce-4208-9b4f-005d200d566b/volumes/kubernetes.io~projected/kube-api-access-pv7t6 DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/231d0cf57d8714e43577c44dbdb66bd96cf2040e27b71c48ae813c5166f4d77b/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d34a9f85672264c2ee4b2649d9e8db0810adbff1a134594fb3f21b1dbf9c524/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7351e46a-1bd0-430b-aec5-1f3111e80370/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/290e7793-2a71-4dfb-9375-be6c7d26d7bb/volumes/kubernetes.io~projected/kube-api-access-gtsj4 DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20d25fae91f0b396d21563561efdfd1b0e72e5696f63ad195d5f21918944a802/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6/volumes/kubernetes.io~projected/kube-api-access-mht6z DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/447a7d8a-1272-419d-a815-6b4353a2011a/volumes/kubernetes.io~projected/kube-api-access-tp5xr DeviceMajor:0 DeviceMinor:330 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:394 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2e70e1-8800-4ab9-9678-34397afb1b9a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c12ce374fa0819b63d7942fb7a52b82ab74d5835c1c2af6dc7b638db0e52f11f/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/896b9a49026b80b09913012a14030199df088847332abe860174c5edf0a30123/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e63f8f3c23e4ef587acc75e0dd1f95048caeb64db9cdb7562623f997e90e70fc/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d83906ba-ac2d-453a-b42b-3a3d958834ee/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:396 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1ef976c87d917640602d55433763af947237fb0c2c8f4f834f01a101a3858b1/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0733f0ad5e5ae264f06ec8aa03f457f50b0123458d2580d34fd80209d4d6a2ae/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cfdaeb4197a9a33728c70254a2094bad01b8bec4ea357f4c752fe1061fa0914/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07/volumes/kubernetes.io~projected/kube-api-access-dfwhh DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36fe958f-eb20-4410-9e52-2a82273fce44/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc284dd-fff4-439d-8732-7b78daf69085/volumes/kubernetes.io~projected/kube-api-access-hwrz8 DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/kube-api-access-lrvcd DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f/volumes/kubernetes.io~projected/kube-api-access-cxrmz DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a11b624f51216074852b68e061ed288d3ee79d5564a9ef1cf3ef9c03713287a9/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24746226-0fdb-40fa-a877-e889fdae7daf/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/508e37556cb72662a36c5f780d4798bd85b0d31d83c83e7fd23212f11a3e0914/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7351e46a-1bd0-430b-aec5-1f3111e80370/volumes/kubernetes.io~projected/kube-api-access-xv75d DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1cefbaa9a0d8c78 MacAddress:2e:ab:09:3a:d5:96 Speed:10000 Mtu:8901} {Name:20d25fae91f0b39 MacAddress:1a:a2:ba:bd:d8:e9 Speed:10000 Mtu:8901} {Name:231d0cf57d8714e MacAddress:ea:20:83:f1:7c:eb Speed:10000 Mtu:8901} {Name:30468e0b0f5f72f MacAddress:b6:e4:d8:dc:c4:e5 Speed:10000 Mtu:8901} {Name:3d34a9f85672264 MacAddress:3a:75:d6:8d:9d:35 Speed:10000 Mtu:8901} {Name:4bcc3d54aa3ef08 MacAddress:ca:bf:3c:e3:c7:c4 Speed:10000 Mtu:8901} {Name:6c71a4b665035f7 MacAddress:ce:be:90:e1:1f:ff Speed:10000 Mtu:8901} {Name:71467ccfa2ab2d5 MacAddress:02:77:49:68:fd:94 Speed:10000 Mtu:8901} {Name:7dd70c2f054731a MacAddress:5e:75:bb:64:31:48 Speed:10000 Mtu:8901} {Name:814f0b4743c0ec6 MacAddress:c2:5b:41:fa:23:64 Speed:10000 Mtu:8901} {Name:8666150bf8975e3 MacAddress:c2:f0:cd:f6:9a:83 Speed:10000 Mtu:8901} {Name:896b9a49026b80b MacAddress:0e:31:fc:eb:84:df Speed:10000 Mtu:8901} {Name:8cfdaeb4197a9a3 MacAddress:96:67:8d:7b:28:54 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ca:8c:2b:05 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:22:89:1f:df:18 Speed:0 Mtu:8901} {Name:c12ce374fa0819b MacAddress:1a:f6:57:65:ca:14 Speed:10000 Mtu:8901} {Name:d7041d0dc8b8f37 MacAddress:5e:98:0a:55:49:f7 Speed:10000 Mtu:8901} {Name:e63f8f3c23e4ef5 MacAddress:de:7c:f7:12:01:eb Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ca:8c:2b:05 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3e:a5:e2:a5:07:95 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:2e:80:73:32:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.495756    5112 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.495799    5112 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496060    5112 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496081    5112 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-122.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496212    5112 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496222    5112 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496232    5112 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496246    5112 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496264    5112 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496427    5112 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496523    5112 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496554    5112 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496583    5112 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496593    5112 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496601    5112 kubelet.go:405] "Adding apiserver pod source"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.496609    5112 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.497427    5112 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.497438    5112 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:10:38.498891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.497888    5112 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 16:10:38.500888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.500868    5112 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 16:10:38.500888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.500885    5112 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 16:10:38.501030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501015    5112 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 16:10:38.501266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501250    5112 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 16:10:38.501331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501290    5112 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 16:10:38.501421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501410    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 16:10:38.501476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501431    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 16:10:38.501476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501442    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 16:10:38.501476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501451    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 16:10:38.501476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501460    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 16:10:38.501476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501470    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501481    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501490    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501501    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501523    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501543    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501559    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501595    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 16:10:38.501702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.501604    5112 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 16:10:38.505109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.505092    5112 server.go:1263] "Started kubelet"
Nov 03 16:10:38.505622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.505188    5112 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 16:10:38.505622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.505217    5112 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 16:10:38.505622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.505284    5112 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 16:10:38.505622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.505619    5112 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 16:10:38.505716 ip-10-0-137-122 systemd[1]: Started Kubernetes Kubelet.
Nov 03 16:10:38.506449 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.506413    5112 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.510469    5112 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.510504    5112 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.510587    5112 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:04:26 +0000 UTC" deadline="2028-08-18 23:12:52.069869484 +0000 UTC"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.510609    5112 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15703h2m13.559264616s"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.510654    5112 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-122.us-west-2.compute.internal\" not found"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.511062    5112 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.510598    5112 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 16:10:38.512030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.511928    5112 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 16:10:38.513759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.513737    5112 factory.go:55] Registering systemd factory
Nov 03 16:10:38.513759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.513761    5112 factory.go:223] Registration of the systemd container factory successfully
Nov 03 16:10:38.514060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514040    5112 factory.go:153] Registering CRI-O factory
Nov 03 16:10:38.514137 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514065    5112 factory.go:223] Registration of the crio container factory successfully
Nov 03 16:10:38.514137 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514119    5112 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 16:10:38.514266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514140    5112 factory.go:103] Registering Raw factory
Nov 03 16:10:38.514266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514154    5112 manager.go:1199] Started watching for new ooms in manager
Nov 03 16:10:38.514513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514499    5112 manager.go:319] Starting recovery of all containers
Nov 03 16:10:38.514925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514868    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets" seLinuxMountContext=""
Nov 03 16:10:38.514990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514929    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 16:10:38.514990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514947    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="203b34a5-c159-485a-83fd-34cf7c348b9d" volumeName="kubernetes.io/projected/203b34a5-c159-485a-83fd-34cf7c348b9d-kube-api-access-57sfl" seLinuxMountContext=""
Nov 03 16:10:38.514990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514961    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f18aa2-9718-4d99-9e5c-0da5f272cd02" volumeName="kubernetes.io/projected/43f18aa2-9718-4d99-9e5c-0da5f272cd02-kube-api-access-frnzm" seLinuxMountContext=""
Nov 03 16:10:38.514990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514970    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2e70e1-8800-4ab9-9678-34397afb1b9a" volumeName="kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 16:10:38.514990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514978    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7351e46a-1bd0-430b-aec5-1f3111e80370" volumeName="kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-tmp" seLinuxMountContext=""
Nov 03 16:10:38.514990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514986    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7351e46a-1bd0-430b-aec5-1f3111e80370" volumeName="kubernetes.io/secret/7351e46a-1bd0-430b-aec5-1f3111e80370-serving-cert" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.514993    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515004    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36fe958f-eb20-4410-9e52-2a82273fce44" volumeName="kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-default-certificate" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515012    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43e7bf8b-0e8b-46e5-a55a-12dd965a41d0" volumeName="kubernetes.io/secret/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-serving-cert" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515020    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515035    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75bf77f-4e2f-43d3-aa27-1ee258dc4db6" volumeName="kubernetes.io/secret/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-cert" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515042    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be48d695-434e-473c-a500-a1e2e8c5de24" volumeName="kubernetes.io/projected/be48d695-434e-473c-a500-a1e2e8c5de24-kube-api-access-wmjrh" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515049    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83906ba-ac2d-453a-b42b-3a3d958834ee" volumeName="kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515059    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" volumeName="kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515066    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" volumeName="kubernetes.io/projected/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-kube-api-access-nttb7" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515076    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515085    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7351e46a-1bd0-430b-aec5-1f3111e80370" volumeName="kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-snapshots" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515092    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515098    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515106    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" volumeName="kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515113    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="203b34a5-c159-485a-83fd-34cf7c348b9d" volumeName="kubernetes.io/configmap/203b34a5-c159-485a-83fd-34cf7c348b9d-serviceca" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515120    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36fe958f-eb20-4410-9e52-2a82273fce44" volumeName="kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515127    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f18aa2-9718-4d99-9e5c-0da5f272cd02" volumeName="kubernetes.io/configmap/43f18aa2-9718-4d99-9e5c-0da5f272cd02-iptables-alerter-script" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515137    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290e7793-2a71-4dfb-9375-be6c7d26d7bb" volumeName="kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-config" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515148    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ff34aa0-6133-41ff-b88a-f5360d6d2f90" volumeName="kubernetes.io/empty-dir/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-tmp-dir" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515157    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515169    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b567c0b7-83c8-46f2-8e94-6ab1ccbffb07" volumeName="kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-tuned" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515189    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c47cd951-b547-4a56-b2c2-1f28b1358ed6" volumeName="kubernetes.io/projected/c47cd951-b547-4a56-b2c2-1f28b1358ed6-kube-api-access-rvcbz" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515213    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83906ba-ac2d-453a-b42b-3a3d958834ee" volumeName="kubernetes.io/projected/d83906ba-ac2d-453a-b42b-3a3d958834ee-kube-api-access-w4lrh" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515224    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" volumeName="kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515233    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754fed8d-9c7a-4cee-8806-794ed3330bc0" volumeName="kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515241    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d79060-9c22-412b-b6df-576c7a2dd340" volumeName="kubernetes.io/configmap/d5d79060-9c22-412b-b6df-576c7a2dd340-konnectivity-ca" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515249    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" volumeName="kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-binary-copy" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515256    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc284dd-fff4-439d-8732-7b78daf69085" volumeName="kubernetes.io/projected/0dc284dd-fff4-439d-8732-7b78daf69085-kube-api-access-hwrz8" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515264    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43e7bf8b-0e8b-46e5-a55a-12dd965a41d0" volumeName="kubernetes.io/configmap/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-config" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515272    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" volumeName="kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515279    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" volumeName="kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515288    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754fed8d-9c7a-4cee-8806-794ed3330bc0" volumeName="kubernetes.io/projected/754fed8d-9c7a-4cee-8806-794ed3330bc0-kube-api-access-7tv8f" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515295    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a75bf77f-4e2f-43d3-aa27-1ee258dc4db6" volumeName="kubernetes.io/projected/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-kube-api-access-mht6z" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515303    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef6cea18-3a39-4c4b-8fba-0284ca6f2e98" volumeName="kubernetes.io/configmap/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-telemetry-config" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515311    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" volumeName="kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 16:10:38.515301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515320    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515328    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24746226-0fdb-40fa-a877-e889fdae7daf" volumeName="kubernetes.io/configmap/24746226-0fdb-40fa-a877-e889fdae7daf-signing-cabundle" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515348    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290e7793-2a71-4dfb-9375-be6c7d26d7bb" volumeName="kubernetes.io/projected/290e7793-2a71-4dfb-9375-be6c7d26d7bb-kube-api-access-gtsj4" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515360    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c078e5-03f1-4b2e-ba52-5c10d58a5d82" volumeName="kubernetes.io/projected/31c078e5-03f1-4b2e-ba52-5c10d58a5d82-kube-api-access-6khfm" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515368    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36fe958f-eb20-4410-9e52-2a82273fce44" volumeName="kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515376    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36fe958f-eb20-4410-9e52-2a82273fce44" volumeName="kubernetes.io/projected/36fe958f-eb20-4410-9e52-2a82273fce44-kube-api-access-q4vrz" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515385    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2e70e1-8800-4ab9-9678-34397afb1b9a" volumeName="kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-config" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515393    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2e70e1-8800-4ab9-9678-34397afb1b9a" volumeName="kubernetes.io/projected/6f2e70e1-8800-4ab9-9678-34397afb1b9a-kube-api-access-6qpk4" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515401    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515409    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515416    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515423    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515430    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a16b1c0a-5c06-4fef-9a5a-664d92a24a25" volumeName="kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515437    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5c0882b-45cb-47a7-983e-01e5748ffab9" volumeName="kubernetes.io/projected/b5c0882b-45cb-47a7-983e-01e5748ffab9-kube-api-access-zg4hz" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515465    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d671aa4a-c3a4-474d-8b2c-9b60ae49c0de" volumeName="kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515473    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36fe958f-eb20-4410-9e52-2a82273fce44" volumeName="kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-stats-auth" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515488    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43e7bf8b-0e8b-46e5-a55a-12dd965a41d0" volumeName="kubernetes.io/projected/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-kube-api-access-k7jzc" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515495    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754fed8d-9c7a-4cee-8806-794ed3330bc0" volumeName="kubernetes.io/configmap/754fed8d-9c7a-4cee-8806-794ed3330bc0-config-volume" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515502    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515512    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b567c0b7-83c8-46f2-8e94-6ab1ccbffb07" volumeName="kubernetes.io/projected/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-kube-api-access-dfwhh" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515520    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d671aa4a-c3a4-474d-8b2c-9b60ae49c0de" volumeName="kubernetes.io/configmap/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-nginx-conf" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515526    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef6cea18-3a39-4c4b-8fba-0284ca6f2e98" volumeName="kubernetes.io/projected/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-kube-api-access-mtrrq" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515534    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7cd889c-edce-4208-9b4f-005d200d566b" volumeName="kubernetes.io/configmap/f7cd889c-edce-4208-9b4f-005d200d566b-config" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515541    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24746226-0fdb-40fa-a877-e889fdae7daf" volumeName="kubernetes.io/secret/24746226-0fdb-40fa-a877-e889fdae7daf-signing-key" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515549    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2e70e1-8800-4ab9-9678-34397afb1b9a" volumeName="kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-env-overrides" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515556    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2e70e1-8800-4ab9-9678-34397afb1b9a" volumeName="kubernetes.io/secret/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515564    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ff34aa0-6133-41ff-b88a-f5360d6d2f90" volumeName="kubernetes.io/projected/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-kube-api-access-l5p8l" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515572    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef6cea18-3a39-4c4b-8fba-0284ca6f2e98" volumeName="kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515579    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7cd889c-edce-4208-9b4f-005d200d566b" volumeName="kubernetes.io/secret/f7cd889c-edce-4208-9b4f-005d200d566b-serving-cert" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515587    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bfea5a0-fec4-4ddc-a409-57656d6bfb7f" volumeName="kubernetes.io/projected/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kube-api-access-cxrmz" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515595    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290e7793-2a71-4dfb-9375-be6c7d26d7bb" volumeName="kubernetes.io/secret/290e7793-2a71-4dfb-9375-be6c7d26d7bb-serving-cert" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515602    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64a4a725-bb12-428a-a3d0-62622448f577" volumeName="kubernetes.io/projected/64a4a725-bb12-428a-a3d0-62622448f577-kube-api-access-hh9g7" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515611    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf495d7-831a-4266-a508-c12d9968971d" volumeName="kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515619    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1a241b6-335b-4ee8-add5-07a5e71666e5" volumeName="kubernetes.io/projected/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-api-access-h6bwg" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515627    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83906ba-ac2d-453a-b42b-3a3d958834ee" volumeName="kubernetes.io/configmap/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-config" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515634    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7cd889c-edce-4208-9b4f-005d200d566b" volumeName="kubernetes.io/projected/f7cd889c-edce-4208-9b4f-005d200d566b-kube-api-access-pv7t6" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515642    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b0366db-bfff-4d9c-af6e-37b79c084739" volumeName="kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515649    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7351e46a-1bd0-430b-aec5-1f3111e80370" volumeName="kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-service-ca-bundle" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515657    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" volumeName="kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515665    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c47cd951-b547-4a56-b2c2-1f28b1358ed6" volumeName="kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-daemon-config" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515672    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11ba431e-e81b-4dbf-9d65-445a66ff6e07" volumeName="kubernetes.io/projected/11ba431e-e81b-4dbf-9d65-445a66ff6e07-kube-api-access-4w872" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515680    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290e7793-2a71-4dfb-9375-be6c7d26d7bb" volumeName="kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-trusted-ca" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515688    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="447a7d8a-1272-419d-a815-6b4353a2011a" volumeName="kubernetes.io/projected/447a7d8a-1272-419d-a815-6b4353a2011a-kube-api-access-tp5xr" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515696    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47333977-43b6-47af-ab13-b5dbee646be5" volumeName="kubernetes.io/projected/47333977-43b6-47af-ab13-b5dbee646be5-kube-api-access-79s9t" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515704    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7351e46a-1bd0-430b-aec5-1f3111e80370" volumeName="kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515712    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="754fed8d-9c7a-4cee-8806-794ed3330bc0" volumeName="kubernetes.io/empty-dir/754fed8d-9c7a-4cee-8806-794ed3330bc0-tmp-dir" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515720    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d79060-9c22-412b-b6df-576c7a2dd340" volumeName="kubernetes.io/secret/d5d79060-9c22-412b-b6df-576c7a2dd340-agent-certs" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515731    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24746226-0fdb-40fa-a877-e889fdae7daf" volumeName="kubernetes.io/projected/24746226-0fdb-40fa-a877-e889fdae7daf-kube-api-access-h6n52" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515741    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7351e46a-1bd0-430b-aec5-1f3111e80370" volumeName="kubernetes.io/projected/7351e46a-1bd0-430b-aec5-1f3111e80370-kube-api-access-xv75d" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515749    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1a241b6-335b-4ee8-add5-07a5e71666e5" volumeName="kubernetes.io/empty-dir/b1a241b6-335b-4ee8-add5-07a5e71666e5-data-volume" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515758    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b567c0b7-83c8-46f2-8e94-6ab1ccbffb07" volumeName="kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-tmp" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515765    5112 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c47cd951-b547-4a56-b2c2-1f28b1358ed6" volumeName="kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cni-binary-copy" seLinuxMountContext=""
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515773    5112 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 16:10:38.517007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.515779    5112 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 16:10:38.533353 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.532900    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:10:38.534440 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.534415    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:10:38.539024 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.538986    5112 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 16:10:38.541332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.541309    5112 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 16:10:38.541332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.541335    5112 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 16:10:38.541533 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.541357    5112 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 16:10:38.541533 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.541396    5112 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 16:10:38.564052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.564031    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:10:38.564174 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.564057    5112 generic.go:356] "Generic (PLEG): container finished" podID="290e7793-2a71-4dfb-9375-be6c7d26d7bb" containerID="5baeab839e0d08d18a7338066961f762077bc9655868802f6ab04a19c2afbe98" exitCode=255
Nov 03 16:10:38.580625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.580565    5112 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="959e8721a5f76ac90915045eada8912ce0c779512dc71c79855defb7f8c20cdf" exitCode=0
Nov 03 16:10:38.580625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.580587    5112 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="fe6f63e6ed661f5fe16dda6d175d4045ba02a3e993544db1bf746d25fed72c81" exitCode=0
Nov 03 16:10:38.580625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.580593    5112 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="b6af3c39d249cefab412729deca5c58340cb4ed54628479f649a79bc754e9f84" exitCode=0
Nov 03 16:10:38.580625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.580598    5112 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="c3a1c8f8f00a9e4cff6781e275bdabf32a056c8a44a1b70024be3c038e15bdf4" exitCode=0
Nov 03 16:10:38.580625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.580603    5112 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="55d8c231f4beb8128c373aa93699cee773a930b362ee63b27c4c556065cefde0" exitCode=0
Nov 03 16:10:38.580625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.580608    5112 generic.go:356] "Generic (PLEG): container finished" podID="f90ec2f4-8ef6-40ec-b47d-c109711dc0ff" containerID="b92dc18c7fcd25022d0daecc74de453079167d32c321b0f60188a135cc323021" exitCode=0
Nov 03 16:10:38.584964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.584932    5112 generic.go:356] "Generic (PLEG): container finished" podID="f1fe4bd879ae26257e3dc2e6b3fda856" containerID="8cb78d9b249a389080e24ca0d0ef5eee947f52af3f29cdaa55f0c58e1ad6d38b" exitCode=0
Nov 03 16:10:38.610729 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.610701    5112 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-122.us-west-2.compute.internal\" not found"
Nov 03 16:10:38.611840 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.611825    5112 manager.go:324] Recovery completed
Nov 03 16:10:38.628676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.628660    5112 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:10:38.630708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.630686    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:10:38.630788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.630725    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:10:38.630788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.630739    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:10:38.633789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633764    5112 cpu_manager.go:226] "Starting" policy="none"
Nov 03 16:10:38.633789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633778    5112 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 16:10:38.633789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633783    5112 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633797    5112 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633944    5112 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633959    5112 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633984    5112 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.633993    5112 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.634001    5112 policy_none.go:50] "Start"
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.634012    5112 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 16:10:38.634007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.634025    5112 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 16:10:38.634662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.634644    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:10:38.634792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.634674    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:10:38.634792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.634687    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:10:38.641801 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.641785    5112 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 16:10:38.651621 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.651607    5112 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 16:10:38.651677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.651623    5112 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 16:10:38.651677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.651630    5112 policy_none.go:44] "Start"
Nov 03 16:10:38.656714 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.656685    5112 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 16:10:38.660609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.660596    5112 manager.go:341] "Starting Device Plugin manager"
Nov 03 16:10:38.660695 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.660628    5112 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 16:10:38.660695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.660636    5112 server.go:87] "Starting device plugin registration server"
Nov 03 16:10:38.660943 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.660930    5112 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 16:10:38.660995 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.660943    5112 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 16:10:38.661042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.661024    5112 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 16:10:38.661165 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.661130    5112 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 16:10:38.661232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.661177    5112 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 16:10:38.661232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.661184    5112 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 16:10:38.662595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.662577    5112 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 16:10:38.662672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.662601    5112 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 16:10:38.662851 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.661249    5112 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:10:38.661163109Z","UUID":"879ae7c2-9052-4b29-a704-0d7602adc9f4","Handler":null,"Name":"","Endpoint":""}
Nov 03 16:10:38.662921 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.662859    5112 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 16:10:38.662921 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.662895    5112 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-122.us-west-2.compute.internal\" not found"
Nov 03 16:10:38.695506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.695485    5112 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:10:38.695595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.695561    5112 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 16:10:38.698071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.698058    5112 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:10:38.698109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.698070    5112 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:10:38.745342 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.745317    5112 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 16:10:38.761624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.761608    5112 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 16:10:38.762603 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.762587    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 16:10:38.762695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.762614    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 16:10:38.762695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.762624    5112 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-122.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 16:10:38.762695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.762669    5112 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.842660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.842591    5112 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"]
Nov 03 16:10:38.843288 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.843255    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"d9d1cb113f9a0ff39835d34e4e74ebc1","Type":"ContainerStarted","Data":"836e1ea7893f7572df8280a0a7c58c039a0a431a8f33e326e161a446d6dc28fb"}
Nov 03 16:10:38.843361 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.843300    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"d9d1cb113f9a0ff39835d34e4e74ebc1","Type":"ContainerStarted","Data":"c6af7087907889f491f2f9f022bd66b1c01dac4a3cba815c2415b72dffdcd510"}
Nov 03 16:10:38.843361 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.843280    5112 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.843361 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.843336    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"f1fe4bd879ae26257e3dc2e6b3fda856","Type":"ContainerStarted","Data":"cb4d432a41f4fd82f963595079f4a6526a32a6b4ae37cb4f957bf334f9bcbd7f"}
Nov 03 16:10:38.843361 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.843348    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"f1fe4bd879ae26257e3dc2e6b3fda856","Type":"ContainerDied","Data":"8cb78d9b249a389080e24ca0d0ef5eee947f52af3f29cdaa55f0c58e1ad6d38b"}
Nov 03 16:10:38.843361 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.843361    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal" event={"ID":"f1fe4bd879ae26257e3dc2e6b3fda856","Type":"ContainerStarted","Data":"99bd7d1c16e2f048a80e7237d46618ad73b67600a5786df18539c026759a5954"}
Nov 03 16:10:38.843610 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.843525    5112 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.869447 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.869428    5112 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.869521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.869482    5112 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.905716 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.905696    5112 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:10:38.905816 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.905730    5112 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.907214 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.907184    5112 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:10:38.907287 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:38.907245    5112 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.997457 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.997411    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.997597 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.997510    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:38.997597 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:38.997540    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9d1cb113f9a0ff39835d34e4e74ebc1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"d9d1cb113f9a0ff39835d34e4e74ebc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.098176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.098105    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.098176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.098118    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.098176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.098148    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.098392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.098183    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9d1cb113f9a0ff39835d34e4e74ebc1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"d9d1cb113f9a0ff39835d34e4e74ebc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.098392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.098214    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1fe4bd879ae26257e3dc2e6b3fda856-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"f1fe4bd879ae26257e3dc2e6b3fda856\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.098392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.098252    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9d1cb113f9a0ff39835d34e4e74ebc1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" (UID: \"d9d1cb113f9a0ff39835d34e4e74ebc1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.497505 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.497470    5112 apiserver.go:52] "Watching apiserver"
Nov 03 16:10:39.602059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.602030    5112 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 16:10:39.603285 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.603250    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cff6v","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-dl567","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7","openshift-console-operator/console-operator-7cd56b8dbc-l2th2","openshift-dns/node-resolver-7nqs4","openshift-insights/insights-runtime-extractor-8szwq","kube-system/global-pull-secret-syncer-h7sv8","openshift-image-registry/image-registry-687cd8bf85-rh7wb","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s","openshift-multus/multus-58crm","kube-system/kas-connection-checker-685895c7c9-6l6l8","openshift-insights/periodic-gathering-xxtp9-hcmjv","openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p","kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal","openshift-insights/insights-operator-7c4986d449-l75v9","openshift-dns/dns-default-rwpz8","openshift-network-diagnostics/network-check-target-cs98m","openshift-ingress-canary/ingress-canary-v4tnz","openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b","openshift-image-registry/node-ca-mxwfz","openshift-network-diagnostics/network-check-source-6f78d59f99-65942","kube-system/kas-connection-checker-685895c7c9-l9r4n","openshift-ovn-kubernetes/ovnkube-node-ljp5j","openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh","openshift-multus/network-metrics-daemon-qd8hz","openshift-cluster-node-tuning-operator/tuned-mlq76","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b","openshift-service-ca/service-ca-5fffc97bdf-7qth8","openshift-ingress/router-default-689b56979-4v24f","openshift-multus/multus-additional-cni-plugins-b7rgl","openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h","openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh","openshift-network-console/networking-console-plugin-86c478bb88-z4krf","kube-system/konnectivity-agent-sg6b8"]
Nov 03 16:10:39.603614 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.603601    5112 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.603739 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.603724    5112 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.604930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.604902    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.606280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.606262    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.606889 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.606871    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.607251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.607235    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.607888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.607866    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:39.610058 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.610041    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:39.616954 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.616917    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.617910 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.617888    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.619465 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.619448    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.655402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655382    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.655516 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655458    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.655516 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655511    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 16:10:39.655602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655550    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 16:10:39.655602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655555    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 16:10:39.655602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655554    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.655602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655520    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.655602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655599    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.655740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655623    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.655944 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.655931    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 16:10:39.656037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.656025    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 16:10:39.656176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.656131    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 16:10:39.656176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.656149    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ggldv\""
Nov 03 16:10:39.656176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.656164    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 16:10:39.656358 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.656165    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-77nn9\""
Nov 03 16:10:39.656358 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.656241    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x47t4\""
Nov 03 16:10:39.661365 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.661350    5112 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:10:39.661414 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:39.661390    5112 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.662930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.662917    5112 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 16:10:39.662968 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:39.662950    5112 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-122.us-west-2.compute.internal"
Nov 03 16:10:39.663990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.663975    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 16:10:39.664106 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.663994    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 16:10:39.664166 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.664143    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 16:10:39.664370 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.664355    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.664619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.664595    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 16:10:39.664706 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.664627    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.665002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.664984    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.665002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.664999    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 16:10:39.665267 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665249    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hc4mx\""
Nov 03 16:10:39.665267 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665263    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.665372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665311    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.665372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665314    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.665372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665338    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cqnft\""
Nov 03 16:10:39.665372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665340    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 16:10:39.665372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665350    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.665372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665360    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zmbll\""
Nov 03 16:10:39.665630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665436    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 16:10:39.665630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665472    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.665630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665484    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.665630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665497    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.665630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665522    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.665630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665608    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kqc54\""
Nov 03 16:10:39.665818 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.665660    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vtfh5\""
Nov 03 16:10:39.702064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702043    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-conf-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.702152 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702072    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.702152 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702092    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.702152 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702107    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.702330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702153    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.702330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702186    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.702330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702226    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-registration-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.702330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702234    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.702330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702270    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-device-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.702330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702294    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-bin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.702625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702421    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-multus-certs\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.702625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702464    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.702625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702501    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-systemd\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.702625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702538    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-sys\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.702625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702567    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-lib-modules\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.702839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702642    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vrz\" (UniqueName: \"kubernetes.io/projected/36fe958f-eb20-4410-9e52-2a82273fce44-kube-api-access-q4vrz\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.702839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702678    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-system-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.702839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702700    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/11ba431e-e81b-4dbf-9d65-445a66ff6e07-kube-api-access-4w872\") pod \"kas-connection-checker-685895c7c9-l9r4n\" (UID: \"11ba431e-e81b-4dbf-9d65-445a66ff6e07\") " pod="kube-system/kas-connection-checker-685895c7c9-l9r4n"
Nov 03 16:10:39.702839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702734    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-kubernetes\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.702982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702887    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.702982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702919    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-tuned\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.702982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702947    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79s9t\" (UniqueName: \"kubernetes.io/projected/47333977-43b6-47af-ab13-b5dbee646be5-kube-api-access-79s9t\") pod \"kas-connection-checker-685895c7c9-cff6v\" (UID: \"47333977-43b6-47af-ab13-b5dbee646be5\") " pod="kube-system/kas-connection-checker-685895c7c9-cff6v"
Nov 03 16:10:39.703128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.702988    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-netns\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.703128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703026    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:39.703128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703039    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-tuned\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.703128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703056    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.703344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703158    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv75d\" (UniqueName: \"kubernetes.io/projected/7351e46a-1bd0-430b-aec5-1f3111e80370-kube-api-access-xv75d\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.703344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703183    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.703344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703257    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:39.703344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703324    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9g7\" (UniqueName: \"kubernetes.io/projected/64a4a725-bb12-428a-a3d0-62622448f577-kube-api-access-hh9g7\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:39.703529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703357    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.703529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703397    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-sys-fs\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.703529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703425    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.703529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703501    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwrz8\" (UniqueName: \"kubernetes.io/projected/0dc284dd-fff4-439d-8732-7b78daf69085-kube-api-access-hwrz8\") pod \"kas-connection-checker-685895c7c9-6l6l8\" (UID: \"0dc284dd-fff4-439d-8732-7b78daf69085\") " pod="kube-system/kas-connection-checker-685895c7c9-6l6l8"
Nov 03 16:10:39.703696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703540    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d79060-9c22-412b-b6df-576c7a2dd340-agent-certs\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:10:39.703696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703589    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjrh\" (UniqueName: \"kubernetes.io/projected/be48d695-434e-473c-a500-a1e2e8c5de24-kube-api-access-wmjrh\") pod \"volume-data-source-validator-58c9f5f46d-kkwbq\" (UID: \"be48d695-434e-473c-a500-a1e2e8c5de24\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"
Nov 03 16:10:39.703696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703623    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7jzc\" (UniqueName: \"kubernetes.io/projected/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-kube-api-access-k7jzc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:39.703696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703656    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.703696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703689    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-service-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.703926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703724    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-multus\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.703926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703761    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtsj4\" (UniqueName: \"kubernetes.io/projected/290e7793-2a71-4dfb-9375-be6c7d26d7bb-kube-api-access-gtsj4\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.703926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703795    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7t6\" (UniqueName: \"kubernetes.io/projected/f7cd889c-edce-4208-9b4f-005d200d566b-kube-api-access-pv7t6\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:39.703926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703856    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nttb7\" (UniqueName: \"kubernetes.io/projected/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-kube-api-access-nttb7\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.703926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703900    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-var-lib-kubelet\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703938    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-service-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703963    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d79060-9c22-412b-b6df-576c7a2dd340-konnectivity-ca\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.703996    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-metrics-certs\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704008    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7351e46a-1bd0-430b-aec5-1f3111e80370-serving-cert\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704043    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704089    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704120    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7cd889c-edce-4208-9b4f-005d200d566b-serving-cert\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:39.704161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704158    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khfm\" (UniqueName: \"kubernetes.io/projected/31c078e5-03f1-4b2e-ba52-5c10d58a5d82-kube-api-access-6khfm\") pod \"network-check-source-6f78d59f99-65942\" (UID: \"31c078e5-03f1-4b2e-ba52-5c10d58a5d82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704188    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-run\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704231    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7351e46a-1bd0-430b-aec5-1f3111e80370-serving-cert\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704329    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-host\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704375    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-default-certificate\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704388    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704393    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7cd889c-edce-4208-9b4f-005d200d566b-serving-cert\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704401    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-stats-auth\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704430    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-tmp\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704467    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-config\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704487    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7cd889c-edce-4208-9b4f-005d200d566b-config\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704505    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysconfig\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704542    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:39.704569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704534    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-tmp\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704577    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-stats-auth\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704591    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-socket-dir-parent\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704617    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/36fe958f-eb20-4410-9e52-2a82273fce44-default-certificate\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704626    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-daemon-config\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704685    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-config\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704690    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704722    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704737    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7cd889c-edce-4208-9b4f-005d200d566b-config\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704774    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704810    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704847    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704872    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cni-binary-copy\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704901    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-etc-kubernetes\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704924    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/290e7793-2a71-4dfb-9375-be6c7d26d7bb-serving-cert\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704940    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.705042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.704956    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtrrq\" (UniqueName: \"kubernetes.io/projected/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-kube-api-access-mtrrq\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705107    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-conf\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705128    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-hostroot\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705143    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705161    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705184    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sfl\" (UniqueName: \"kubernetes.io/projected/203b34a5-c159-485a-83fd-34cf7c348b9d-kube-api-access-57sfl\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705261    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705336    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-kubelet-config\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705354    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/290e7793-2a71-4dfb-9375-be6c7d26d7bb-serving-cert\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705385    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-kubelet\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705418    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705436    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705476    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-tmp\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705527    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705557    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705623    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-tmp\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705621    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lrh\" (UniqueName: \"kubernetes.io/projected/d83906ba-ac2d-453a-b42b-3a3d958834ee-kube-api-access-w4lrh\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705659    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-socket-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705681    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrmz\" (UniqueName: \"kubernetes.io/projected/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kube-api-access-cxrmz\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705697    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/203b34a5-c159-485a-83fd-34cf7c348b9d-serviceca\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705730    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cnibin\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705737    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36fe958f-eb20-4410-9e52-2a82273fce44-service-ca-bundle\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705746    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.705754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705762    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwhh\" (UniqueName: \"kubernetes.io/projected/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-kube-api-access-dfwhh\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705781    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-dbus\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705798    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-modprobe-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705850    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-k8s-cni-cncf-io\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705868    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcbz\" (UniqueName: \"kubernetes.io/projected/c47cd951-b547-4a56-b2c2-1f28b1358ed6-kube-api-access-rvcbz\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705893    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsqfl\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705909    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/203b34a5-c159-485a-83fd-34cf7c348b9d-host\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705929    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705945    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705970    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.705995    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-snapshots\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.706044    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d83906ba-ac2d-453a-b42b-3a3d958834ee-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.706066    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cnibin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.706713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.706123    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7351e46a-1bd0-430b-aec5-1f3111e80370-snapshots\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.711190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711166    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 16:10:39.711190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711181    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 16:10:39.711392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711233    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.711392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711290    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 16:10:39.711392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711308    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.711550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711434    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 16:10:39.711550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711444    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.711550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711534    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711562    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711596    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711612    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711621    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711614    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711629    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dgbbj\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711667    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711596    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 16:10:39.711696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711617    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.712021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711726    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 16:10:39.712021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711799    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxk2h\""
Nov 03 16:10:39.712021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711827    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 16:10:39.712021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.711976    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 16:10:39.712264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712250    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9gv4t\""
Nov 03 16:10:39.712314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712304    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.712361 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712321    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rqr4v\""
Nov 03 16:10:39.712361 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712332    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.712509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712356    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 16:10:39.712509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712431    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7zlt\""
Nov 03 16:10:39.712509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712439    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 16:10:39.712509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712487    5112 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 16:10:39.712639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712565    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.712639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712492    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.712639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.712592    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4pfhl\""
Nov 03 16:10:39.714506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.713666    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.714506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.713718    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 16:10:39.714506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.713773    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.714506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.713841    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.714506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.713966    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wnjzm\""
Nov 03 16:10:39.714506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.714389    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 16:10:39.714506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.714484    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 16:10:39.715163 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.715139    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.715163 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.715159    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7351e46a-1bd0-430b-aec5-1f3111e80370-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.715582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.715542    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d79060-9c22-412b-b6df-576c7a2dd340-konnectivity-ca\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:10:39.715768 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.715750    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:39.715973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.715954    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 16:10:39.716105 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.716068    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 16:10:39.716423 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.716388    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gswn6\""
Nov 03 16:10:39.716517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.716499    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4xv66\""
Nov 03 16:10:39.717238 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.717217    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.717331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.717316    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/290e7793-2a71-4dfb-9375-be6c7d26d7bb-trusted-ca\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.718102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.717515    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0366db-bfff-4d9c-af6e-37b79c084739-original-pull-secret\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.718579 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718555    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 16:10:39.718688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718575    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x72k5\""
Nov 03 16:10:39.718688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718587    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 16:10:39.718688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718598    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.718688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718642    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 16:10:39.718688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718643    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 16:10:39.718688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718650    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.718688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718670    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718724    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718740    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4vrm9\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718753    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718778    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718789    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718816    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718823    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718819    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718849    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718898    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xzbs4\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718852    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zz2z\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718850    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.718990    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cjszs\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719016    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 16:10:39.719064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719025    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 16:10:39.719595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719112    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xhvl5\""
Nov 03 16:10:39.719595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719161    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 16:10:39.719595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719285    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zsc2l\""
Nov 03 16:10:39.719595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719560    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kkrgk\""
Nov 03 16:10:39.719716 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719628    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tk7qx\""
Nov 03 16:10:39.719752 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719714    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 16:10:39.719814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719802    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mvkm5\""
Nov 03 16:10:39.719892 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.719883    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p6cn5\""
Nov 03 16:10:39.720165 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.720151    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 16:10:39.720218 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.720165    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 16:10:39.722635 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.722614    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.723587 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.723570    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.724711 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.724694    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.724778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.724760    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d79060-9c22-412b-b6df-576c7a2dd340-agent-certs\") pod \"konnectivity-agent-sg6b8\" (UID: \"d5d79060-9c22-412b-b6df-576c7a2dd340\") " pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:10:39.724948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.724930    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-daemon-config\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.725104 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.725088    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cni-binary-copy\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.725104 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.725098    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.725277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.725260    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.725923 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.725908    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d671aa4a-c3a4-474d-8b2c-9b60ae49c0de-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z4krf\" (UID: \"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:39.726341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.726326    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/203b34a5-c159-485a-83fd-34cf7c348b9d-serviceca\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:10:39.726481 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.726468    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.726590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.726572    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.758331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.758286    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.758409 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.758335    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9g7\" (UniqueName: \"kubernetes.io/projected/64a4a725-bb12-428a-a3d0-62622448f577-kube-api-access-hh9g7\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:39.758409 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.758335    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79s9t\" (UniqueName: \"kubernetes.io/projected/47333977-43b6-47af-ab13-b5dbee646be5-kube-api-access-79s9t\") pod \"kas-connection-checker-685895c7c9-cff6v\" (UID: \"47333977-43b6-47af-ab13-b5dbee646be5\") " pod="kube-system/kas-connection-checker-685895c7c9-cff6v"
Nov 03 16:10:39.758520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.758501    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwrz8\" (UniqueName: \"kubernetes.io/projected/0dc284dd-fff4-439d-8732-7b78daf69085-kube-api-access-hwrz8\") pod \"kas-connection-checker-685895c7c9-6l6l8\" (UID: \"0dc284dd-fff4-439d-8732-7b78daf69085\") " pod="kube-system/kas-connection-checker-685895c7c9-6l6l8"
Nov 03 16:10:39.758713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.758694    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vrz\" (UniqueName: \"kubernetes.io/projected/36fe958f-eb20-4410-9e52-2a82273fce44-kube-api-access-q4vrz\") pod \"router-default-689b56979-4v24f\" (UID: \"36fe958f-eb20-4410-9e52-2a82273fce44\") " pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.759294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.759278    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv75d\" (UniqueName: \"kubernetes.io/projected/7351e46a-1bd0-430b-aec5-1f3111e80370-kube-api-access-xv75d\") pod \"insights-operator-7c4986d449-l75v9\" (UID: \"7351e46a-1bd0-430b-aec5-1f3111e80370\") " pod="openshift-insights/insights-operator-7c4986d449-l75v9"
Nov 03 16:10:39.759411 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.759393    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w872\" (UniqueName: \"kubernetes.io/projected/11ba431e-e81b-4dbf-9d65-445a66ff6e07-kube-api-access-4w872\") pod \"kas-connection-checker-685895c7c9-l9r4n\" (UID: \"11ba431e-e81b-4dbf-9d65-445a66ff6e07\") " pod="kube-system/kas-connection-checker-685895c7c9-l9r4n"
Nov 03 16:10:39.763368 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.763346    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwhh\" (UniqueName: \"kubernetes.io/projected/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-kube-api-access-dfwhh\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.763607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.763588    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sfl\" (UniqueName: \"kubernetes.io/projected/203b34a5-c159-485a-83fd-34cf7c348b9d-kube-api-access-57sfl\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:10:39.763721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.763706    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lrh\" (UniqueName: \"kubernetes.io/projected/d83906ba-ac2d-453a-b42b-3a3d958834ee-kube-api-access-w4lrh\") pod \"cluster-samples-operator-6ddb6c8b94-8vh2s\" (UID: \"d83906ba-ac2d-453a-b42b-3a3d958834ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.768400 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.767838    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nttb7\" (UniqueName: \"kubernetes.io/projected/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-kube-api-access-nttb7\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.768400 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768231    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpljr\" (UniqueName: \"kubernetes.io/projected/a16b1c0a-5c06-4fef-9a5a-664d92a24a25-kube-api-access-zpljr\") pod \"network-check-target-cs98m\" (UID: \"a16b1c0a-5c06-4fef-9a5a-664d92a24a25\") " pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:39.768400 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768244    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtsj4\" (UniqueName: \"kubernetes.io/projected/290e7793-2a71-4dfb-9375-be6c7d26d7bb-kube-api-access-gtsj4\") pod \"console-operator-7cd56b8dbc-l2th2\" (UID: \"290e7793-2a71-4dfb-9375-be6c7d26d7bb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:39.768400 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768310    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsqfl\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\") pod \"image-registry-6f4bdbf4dd-bqs7h\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") " pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.768400 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768350    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrmz\" (UniqueName: \"kubernetes.io/projected/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kube-api-access-cxrmz\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.768608 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768400    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7jzc\" (UniqueName: \"kubernetes.io/projected/43e7bf8b-0e8b-46e5-a55a-12dd965a41d0-kube-api-access-k7jzc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b\" (UID: \"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b"
Nov 03 16:10:39.768608 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768415    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjrh\" (UniqueName: \"kubernetes.io/projected/be48d695-434e-473c-a500-a1e2e8c5de24-kube-api-access-wmjrh\") pod \"volume-data-source-validator-58c9f5f46d-kkwbq\" (UID: \"be48d695-434e-473c-a500-a1e2e8c5de24\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kkwbq"
Nov 03 16:10:39.768608 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768469    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcbz\" (UniqueName: \"kubernetes.io/projected/c47cd951-b547-4a56-b2c2-1f28b1358ed6-kube-api-access-rvcbz\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.768797 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768776    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7t6\" (UniqueName: \"kubernetes.io/projected/f7cd889c-edce-4208-9b4f-005d200d566b-kube-api-access-pv7t6\") pod \"service-ca-operator-544d749c8c-p6mlh\" (UID: \"f7cd889c-edce-4208-9b4f-005d200d566b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh"
Nov 03 16:10:39.768963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.768948    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtrrq\" (UniqueName: \"kubernetes.io/projected/ef6cea18-3a39-4c4b-8fba-0284ca6f2e98-kube-api-access-mtrrq\") pod \"cluster-monitoring-operator-884576c57-jnx8b\" (UID: \"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.769042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.769024    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khfm\" (UniqueName: \"kubernetes.io/projected/31c078e5-03f1-4b2e-ba52-5c10d58a5d82-kube-api-access-6khfm\") pod \"network-check-source-6f78d59f99-65942\" (UID: \"31c078e5-03f1-4b2e-ba52-5c10d58a5d82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-65942"
Nov 03 16:10:39.806873 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.806849    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-var-lib-kubelet\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.806947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.806878    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-cert\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:39.806947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.806894    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mht6z\" (UniqueName: \"kubernetes.io/projected/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-kube-api-access-mht6z\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:39.806947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.806916    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43f18aa2-9718-4d99-9e5c-0da5f272cd02-host-slash\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:10:39.806947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.806937    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-run\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.807121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.806991    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-run\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.807121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.806998    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-var-lib-kubelet\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.807121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807011    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/754fed8d-9c7a-4cee-8806-794ed3330bc0-config-volume\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.807121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807043    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-etc-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.807121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807068    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-socket-dir-parent\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.807121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807093    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807118    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-cert\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807135    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-system-cni-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807140    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-socket-dir-parent\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807165    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-systemd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807220    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpk4\" (UniqueName: \"kubernetes.io/projected/6f2e70e1-8800-4ab9-9678-34397afb1b9a-kube-api-access-6qpk4\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807246    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807270    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807293    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-etc-kubernetes\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807292    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/754fed8d-9c7a-4cee-8806-794ed3330bc0-config-volume\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.807405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807342    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-etc-kubernetes\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807407    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-kubelet\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807428    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-node-log\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807487    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-kubelet-config\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807530    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807539    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-kubelet-config\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807561    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24746226-0fdb-40fa-a877-e889fdae7daf-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807585    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807601    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4hz\" (UniqueName: \"kubernetes.io/projected/b5c0882b-45cb-47a7-983e-01e5748ffab9-kube-api-access-zg4hz\") pod \"migrator-66df5d496b-svr6p\" (UID: \"b5c0882b-45cb-47a7-983e-01e5748ffab9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807612    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807619    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-kubelet\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807660    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-kubelet\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807660    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cnibin\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.807759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807689    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-cnibin\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807787    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/24746226-0fdb-40fa-a877-e889fdae7daf-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807791    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-tmp-dir\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807820    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-k8s-cni-cncf-io\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807855    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-k8s-cni-cncf-io\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807861    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-log-socket\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807885    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cnibin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807906    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-conf-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807864    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-tmp-dir\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807931    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-registration-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807957    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-conf-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807972    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-cnibin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.807987    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovn-node-metrics-cert\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808014    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjk5\" (UniqueName: \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808026    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-registration-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808083    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.808187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808177    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-bin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808193    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-multus-certs\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808242    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-systemd\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808260    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-bin\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808312    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-sys\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808305    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-multus-certs\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808350    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-systemd\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808351    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovn-node-metrics-cert\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808373    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-lib-modules\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808410    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-sys\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808429    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-system-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808492    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-kubernetes\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808517    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-system-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808527    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24746226-0fdb-40fa-a877-e889fdae7daf-signing-key\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808544    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808551    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-kubernetes\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808556    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-lib-modules\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808616    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808630    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-multus\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808649    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754fed8d-9c7a-4cee-8806-794ed3330bc0-tmp-dir\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808663    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-var-lib-cni-multus\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808669    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808701    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-var-lib-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808722    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/24746226-0fdb-40fa-a877-e889fdae7daf-signing-key\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808729    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808772    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754fed8d-9c7a-4cee-8806-794ed3330bc0-tmp-dir\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808769    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808799    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.808811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808804    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808833    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-env-overrides\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808848    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-script-lib\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808861    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-host\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808877    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bwg\" (UniqueName: \"kubernetes.io/projected/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-api-access-h6bwg\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808899    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808916    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysconfig\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808933    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808942    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/754fed8d-9c7a-4cee-8806-794ed3330bc0-metrics-tls\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808958    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808980    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvcd\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.808996    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-netns\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809001    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809018    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-conf\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809107    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-host\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809122    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-conf\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809176    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysconfig\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809224    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-script-lib\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809232    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f90ec2f4-8ef6-40ec-b47d-c109711dc0ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7rgl\" (UID: \"f90ec2f4-8ef6-40ec-b47d-c109711dc0ff\") " pod="openshift-multus/multus-additional-cni-plugins-b7rgl"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809241    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-env-overrides\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809292    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-hostroot\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809367    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-hostroot\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809395    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809421    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tv8f\" (UniqueName: \"kubernetes.io/projected/754fed8d-9c7a-4cee-8806-794ed3330bc0-kube-api-access-7tv8f\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809451    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-ovn\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809475    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809507    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-socket-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809530    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a241b6-335b-4ee8-add5-07a5e71666e5-crio-socket\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809546    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-dbus\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809563    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809606    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809623    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-socket-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809631    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0366db-bfff-4d9c-af6e-37b79c084739-dbus\") pod \"global-pull-secret-syncer-h7sv8\" (UID: \"2b0366db-bfff-4d9c-af6e-37b79c084739\") " pod="kube-system/global-pull-secret-syncer-h7sv8"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809623    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-hosts-file\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809697    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43f18aa2-9718-4d99-9e5c-0da5f272cd02-iptables-alerter-script\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809718    5112 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 16:10:39.809745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809722    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnzm\" (UniqueName: \"kubernetes.io/projected/43f18aa2-9718-4d99-9e5c-0da5f272cd02-kube-api-access-frnzm\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809790    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-modprobe-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809807    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/203b34a5-c159-485a-83fd-34cf7c348b9d-host\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809825    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43f18aa2-9718-4d99-9e5c-0da5f272cd02-iptables-alerter-script\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809842    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-slash\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809864    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/203b34a5-c159-485a-83fd-34cf7c348b9d-host\") pod \"node-ca-mxwfz\" (UID: \"203b34a5-c159-485a-83fd-34cf7c348b9d\") " pod="openshift-image-registry/node-ca-mxwfz"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809866    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809909    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-modprobe-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809908    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-config\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809966    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.809982    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810031    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-device-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810070    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-netd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810096    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5xr\" (UniqueName: \"kubernetes.io/projected/447a7d8a-1272-419d-a815-6b4353a2011a-kube-api-access-tp5xr\") pod \"migrator-66df5d496b-2b5rh\" (UID: \"447a7d8a-1272-419d-a815-6b4353a2011a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810102    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-device-dir\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810123    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-netns\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810157    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5p8l\" (UniqueName: \"kubernetes.io/projected/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-kube-api-access-l5p8l\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810184    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810230    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810258    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-bin\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810272    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-host-run-netns\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810183    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f2e70e1-8800-4ab9-9678-34397afb1b9a-ovnkube-config\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810281    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810342    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47cd951-b547-4a56-b2c2-1f28b1358ed6-multus-cni-dir\") pod \"multus-58crm\" (UID: \"c47cd951-b547-4a56-b2c2-1f28b1358ed6\") " pod="openshift-multus/multus-58crm"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810386    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810410    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-sys-fs\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810420    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b567c0b7-83c8-46f2-8e94-6ab1ccbffb07-etc-sysctl-d\") pod \"tuned-mlq76\" (UID: \"b567c0b7-83c8-46f2-8e94-6ab1ccbffb07\") " pod="openshift-cluster-node-tuning-operator/tuned-mlq76"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810441    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6n52\" (UniqueName: \"kubernetes.io/projected/24746226-0fdb-40fa-a877-e889fdae7daf-kube-api-access-h6n52\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810465    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-sys-fs\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810479    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bfea5a0-fec4-4ddc-a409-57656d6bfb7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxtt7\" (UID: \"0bfea5a0-fec4-4ddc-a409-57656d6bfb7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxtt7"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810507    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-systemd-units\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810573    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810618    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a241b6-335b-4ee8-add5-07a5e71666e5-data-volume\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810643    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810703    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1a241b6-335b-4ee8-add5-07a5e71666e5-data-volume\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.810857    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.811186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.811105    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.812379 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.812128    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a4a725-bb12-428a-a3d0-62622448f577-metrics-certs\") pod \"network-metrics-daemon-qd8hz\" (UID: \"64a4a725-bb12-428a-a3d0-62622448f577\") " pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:39.862414 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.862383    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.863071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863014    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvcd\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\") pod \"image-registry-687cd8bf85-rh7wb\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") " pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.863178 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863089    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjk5\" (UniqueName: \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\") pod \"periodic-gathering-xxtp9-hcmjv\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") " pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:10:39.863350 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863192    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpk4\" (UniqueName: \"kubernetes.io/projected/6f2e70e1-8800-4ab9-9678-34397afb1b9a-kube-api-access-6qpk4\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.863541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863509    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tv8f\" (UniqueName: \"kubernetes.io/projected/754fed8d-9c7a-4cee-8806-794ed3330bc0-kube-api-access-7tv8f\") pod \"dns-default-rwpz8\" (UID: \"754fed8d-9c7a-4cee-8806-794ed3330bc0\") " pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:39.863677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863594    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4hz\" (UniqueName: \"kubernetes.io/projected/b5c0882b-45cb-47a7-983e-01e5748ffab9-kube-api-access-zg4hz\") pod \"migrator-66df5d496b-svr6p\" (UID: \"b5c0882b-45cb-47a7-983e-01e5748ffab9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svr6p"
Nov 03 16:10:39.863892 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863870    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5p8l\" (UniqueName: \"kubernetes.io/projected/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-kube-api-access-l5p8l\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:10:39.863988 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863877    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bwg\" (UniqueName: \"kubernetes.io/projected/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-api-access-h6bwg\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.863988 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.863967    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnzm\" (UniqueName: \"kubernetes.io/projected/43f18aa2-9718-4d99-9e5c-0da5f272cd02-kube-api-access-frnzm\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:10:39.864093 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.864002    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mht6z\" (UniqueName: \"kubernetes.io/projected/a75bf77f-4e2f-43d3-aa27-1ee258dc4db6-kube-api-access-mht6z\") pod \"ingress-canary-v4tnz\" (UID: \"a75bf77f-4e2f-43d3-aa27-1ee258dc4db6\") " pod="openshift-ingress-canary/ingress-canary-v4tnz"
Nov 03 16:10:39.864149 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.864119    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5xr\" (UniqueName: \"kubernetes.io/projected/447a7d8a-1272-419d-a815-6b4353a2011a-kube-api-access-tp5xr\") pod \"migrator-66df5d496b-2b5rh\" (UID: \"447a7d8a-1272-419d-a815-6b4353a2011a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2b5rh"
Nov 03 16:10:39.864336 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.864320    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6n52\" (UniqueName: \"kubernetes.io/projected/24746226-0fdb-40fa-a877-e889fdae7daf-kube-api-access-h6n52\") pod \"service-ca-5fffc97bdf-7qth8\" (UID: \"24746226-0fdb-40fa-a877-e889fdae7daf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qth8"
Nov 03 16:10:39.905366 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.905342    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:39.905569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.905547    5112 scope.go:122] "RemoveContainer" containerID="5baeab839e0d08d18a7338066961f762077bc9655868802f6ab04a19c2afbe98"
Nov 03 16:10:39.906507 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.906490    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"
Nov 03 16:10:39.907312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.907298    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"
Nov 03 16:10:39.907781 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.907762    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:39.908338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.908324    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf"
Nov 03 16:10:39.911088 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911058    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qd8hz"
Nov 03 16:10:39.911237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911219    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-kubelet\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911249    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-node-log\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911266    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911287    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-log-socket\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911313    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911329    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-node-log\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911336    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-var-lib-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911366    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911375    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-var-lib-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911334    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-log-socket\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911334    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911396    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911404    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-kubelet\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911420    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:39.911429    5112 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:39.911450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911442    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-netns\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:39.911476    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:40.411463154 +0000 UTC m=+1.990766994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911487    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-run-netns\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911498    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-ovn\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911522    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a241b6-335b-4ee8-add5-07a5e71666e5-crio-socket\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911538    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911552    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-hosts-file\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:39.911575    5112 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911584    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1a241b6-335b-4ee8-add5-07a5e71666e5-crio-socket\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911594    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911605    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-slash\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911613    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-ovn\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911576    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-slash\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911644    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff34aa0-6133-41ff-b88a-f5360d6d2f90-hosts-file\") pod \"node-resolver-7nqs4\" (UID: \"7ff34aa0-6133-41ff-b88a-f5360d6d2f90\") " pod="openshift-dns/node-resolver-7nqs4"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:39.911647    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:40.411631088 +0000 UTC m=+1.990934938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911666    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-netd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911691    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-bin\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911707    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-netd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911718    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-systemd-units\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911748    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43f18aa2-9718-4d99-9e5c-0da5f272cd02-host-slash\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911751    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-systemd-units\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911759    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-host-cni-bin\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911774    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-etc-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911799    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-systemd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911815    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43f18aa2-9718-4d99-9e5c-0da5f272cd02-host-slash\") pod \"iptables-alerter-dl567\" (UID: \"43f18aa2-9718-4d99-9e5c-0da5f272cd02\") " pod="openshift-network-operator/iptables-alerter-dl567"
Nov 03 16:10:39.911829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911819    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-etc-openvswitch\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.912540 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.911870    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f2e70e1-8800-4ab9-9678-34397afb1b9a-run-systemd\") pod \"ovnkube-node-ljp5j\" (UID: \"6f2e70e1-8800-4ab9-9678-34397afb1b9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:39.917224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.917189    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:39.918332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:39.918273    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:40.399966 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.399937    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s"]
Nov 03 16:10:40.402964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.402916    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-689b56979-4v24f"]
Nov 03 16:10:40.403710 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.403693    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rwpz8"]
Nov 03 16:10:40.425511 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.425308    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:40.425511 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.425353    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:40.425511 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:40.425444    5112 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:40.425682 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:40.425515    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:41.425495209 +0000 UTC m=+3.004799048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:40.425682 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:40.425584    5112 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:40.425682 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:40.425637    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:41.425619659 +0000 UTC m=+3.004923499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:40.428244 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:40.428148    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754fed8d_9c7a_4cee_8806_794ed3330bc0.slice/crio-6787825deeb35d81954ae1c61f9da469109a831ecf0cf454d7f7acc9ce768f97 WatchSource:0}: Error finding container 6787825deeb35d81954ae1c61f9da469109a831ecf0cf454d7f7acc9ce768f97: Status 404 returned error can't find the container with id 6787825deeb35d81954ae1c61f9da469109a831ecf0cf454d7f7acc9ce768f97
Nov 03 16:10:40.430323 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.430307    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:10:40.454143 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.454116    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b"]
Nov 03 16:10:40.458262 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:40.457567    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6cea18_3a39_4c4b_8fba_0284ca6f2e98.slice/crio-fded57cb95bc56d63dd081765837adcdba4242d701d526bddf66564e7a6a8d8d WatchSource:0}: Error finding container fded57cb95bc56d63dd081765837adcdba4242d701d526bddf66564e7a6a8d8d: Status 404 returned error can't find the container with id fded57cb95bc56d63dd081765837adcdba4242d701d526bddf66564e7a6a8d8d
Nov 03 16:10:40.473395 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.473368    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:40.479439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.479416    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qd8hz"]
Nov 03 16:10:40.484473 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:40.484443    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a4a725_bb12_428a_a3d0_62622448f577.slice/crio-7d623680f29394703c22928aa250bf5a0230f32d1a596f8bec7bd4545478d7c8 WatchSource:0}: Error finding container 7d623680f29394703c22928aa250bf5a0230f32d1a596f8bec7bd4545478d7c8: Status 404 returned error can't find the container with id 7d623680f29394703c22928aa250bf5a0230f32d1a596f8bec7bd4545478d7c8
Nov 03 16:10:40.492580 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.492557    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-z4krf"]
Nov 03 16:10:40.502526 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:40.502501    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd671aa4a_c3a4_474d_8b2c_9b60ae49c0de.slice/crio-6a395e0d88dff486993935c6a6b814363c03f8e9705770c26b84241fe5bfd05b WatchSource:0}: Error finding container 6a395e0d88dff486993935c6a6b814363c03f8e9705770c26b84241fe5bfd05b: Status 404 returned error can't find the container with id 6a395e0d88dff486993935c6a6b814363c03f8e9705770c26b84241fe5bfd05b
Nov 03 16:10:40.529104 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.529071    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687cd8bf85-rh7wb"]
Nov 03 16:10:40.531970 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:40.531940    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1ab42f_45fe_4c02_89b8_3fc4aa9fb573.slice/crio-f90782f3bc4e74abe17c3c536942ea0fcc6dff70336fe116741694b3ff1a1f06 WatchSource:0}: Error finding container f90782f3bc4e74abe17c3c536942ea0fcc6dff70336fe116741694b3ff1a1f06: Status 404 returned error can't find the container with id f90782f3bc4e74abe17c3c536942ea0fcc6dff70336fe116741694b3ff1a1f06
Nov 03 16:10:40.566683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.566656    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"]
Nov 03 16:10:40.579258 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:40.579070    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf495d7_831a_4266_a508_c12d9968971d.slice/crio-de057a33fdc0274d400863a00be548874e51d8ef4232b52adeceb1e79fd17777 WatchSource:0}: Error finding container de057a33fdc0274d400863a00be548874e51d8ef4232b52adeceb1e79fd17777: Status 404 returned error can't find the container with id de057a33fdc0274d400863a00be548874e51d8ef4232b52adeceb1e79fd17777
Nov 03 16:10:40.609348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.609314    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd8hz" event={"ID":"64a4a725-bb12-428a-a3d0-62622448f577","Type":"ContainerStarted","Data":"7d623680f29394703c22928aa250bf5a0230f32d1a596f8bec7bd4545478d7c8"}
Nov 03 16:10:40.610421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.610394    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b" event={"ID":"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98","Type":"ContainerStarted","Data":"fded57cb95bc56d63dd081765837adcdba4242d701d526bddf66564e7a6a8d8d"}
Nov 03 16:10:40.612053 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.612027    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h" event={"ID":"9cf495d7-831a-4266-a508-c12d9968971d","Type":"ContainerStarted","Data":"de057a33fdc0274d400863a00be548874e51d8ef4232b52adeceb1e79fd17777"}
Nov 03 16:10:40.614095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.614075    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:10:40.614274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.614179    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2" event={"ID":"290e7793-2a71-4dfb-9375-be6c7d26d7bb","Type":"ContainerStarted","Data":"97b1dd1093e328ccebc6d1393e6b4fedcd23b7bd843d8450fc369af9bebb2787"}
Nov 03 16:10:40.615995 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.615974    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb" event={"ID":"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573","Type":"ContainerStarted","Data":"f90782f3bc4e74abe17c3c536942ea0fcc6dff70336fe116741694b3ff1a1f06"}
Nov 03 16:10:40.617613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.617581    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-689b56979-4v24f" event={"ID":"36fe958f-eb20-4410-9e52-2a82273fce44","Type":"ContainerStarted","Data":"7385bf932eb143a9cf8df90766de587580ff99099b351501ea68ae98fd94f468"}
Nov 03 16:10:40.617698 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.617619    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-689b56979-4v24f" event={"ID":"36fe958f-eb20-4410-9e52-2a82273fce44","Type":"ContainerStarted","Data":"c677d611105a358e911963e2b9e55922de4b78e44cd11a0f912d780c7e22a8ca"}
Nov 03 16:10:40.625004 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.624966    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf" event={"ID":"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de","Type":"ContainerStarted","Data":"6a395e0d88dff486993935c6a6b814363c03f8e9705770c26b84241fe5bfd05b"}
Nov 03 16:10:40.627639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.627607    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwpz8" event={"ID":"754fed8d-9c7a-4cee-8806-794ed3330bc0","Type":"ContainerStarted","Data":"6787825deeb35d81954ae1c61f9da469109a831ecf0cf454d7f7acc9ce768f97"}
Nov 03 16:10:40.628868 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.628832    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s" event={"ID":"d83906ba-ac2d-453a-b42b-3a3d958834ee","Type":"ContainerStarted","Data":"d2498e5b760c6a6c8a8176b5810c7f59b51f6814591004e83a51688e6cad04a5"}
Nov 03 16:10:40.905741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.905664    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:40.908372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.908332    5112 patch_prober.go:29] interesting pod/router-default-689b56979-4v24f container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Nov 03 16:10:40.908372 ip-10-0-137-122 kubenswrapper[5112]: [+]has-synced ok
Nov 03 16:10:40.908372 ip-10-0-137-122 kubenswrapper[5112]: [+]process-running ok
Nov 03 16:10:40.908372 ip-10-0-137-122 kubenswrapper[5112]: healthz check failed
Nov 03 16:10:40.908574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.908396    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-689b56979-4v24f" podUID="36fe958f-eb20-4410-9e52-2a82273fce44" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 03 16:10:40.954185 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.954151    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:40.959505 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:40.959471    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l2th2"
Nov 03 16:10:41.199712 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.199633    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:41.433296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.433262    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:41.433455 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.433303    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:41.433455 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:41.433392    5112 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 16:10:41.433455 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:41.433445    5112 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:41.433593 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:41.433480    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:43.433458266 +0000 UTC m=+5.012762099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : configmap "kube-rbac-proxy" not found
Nov 03 16:10:41.433593 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:41.433501    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:43.433490676 +0000 UTC m=+5.012794515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:41.645089 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.645054    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h" event={"ID":"9cf495d7-831a-4266-a508-c12d9968971d","Type":"ContainerStarted","Data":"f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b"}
Nov 03 16:10:41.645652 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.645597    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:10:41.648109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.648087    5112 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:10:41.648329 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.648309    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb" event={"ID":"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573","Type":"ContainerStarted","Data":"90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649"}
Nov 03 16:10:41.907242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:41.907189    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:42.408854 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:42.408611    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:42.410884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:42.410128    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-689b56979-4v24f"
Nov 03 16:10:42.657055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:42.657020    5112 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 16:10:43.463880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:43.463837    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:43.464060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:43.463884    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:43.464060 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:43.464032    5112 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:43.464174 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:10:43.464119    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls podName:b1a241b6-335b-4ee8-add5-07a5e71666e5 nodeName:}" failed. No retries permitted until 2026-11-03 16:10:47.464099028 +0000 UTC m=+9.043402880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8szwq" (UID: "b1a241b6-335b-4ee8-add5-07a5e71666e5") : secret "insights-runtime-extractor-tls" not found
Nov 03 16:10:43.464377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:43.464353    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1a241b6-335b-4ee8-add5-07a5e71666e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:43.661826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:43.661792    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-z4krf" event={"ID":"d671aa4a-c3a4-474d-8b2c-9b60ae49c0de","Type":"ContainerStarted","Data":"3e5c4b9ed8548fd4de2359aa1e616f42a4c1c086219e9f24b73da8599a224094"}
Nov 03 16:10:43.663497 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:43.663467    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd8hz" event={"ID":"64a4a725-bb12-428a-a3d0-62622448f577","Type":"ContainerStarted","Data":"bc7b0c348301b6273d5a9894be133da0d887405a8a88c406a8f71fb1174b8b48"}
Nov 03 16:10:44.669658 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:44.669626    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwpz8" event={"ID":"754fed8d-9c7a-4cee-8806-794ed3330bc0","Type":"ContainerStarted","Data":"2a95a347444466cc75d54eb2207331a1bb15ff4b76538b51cfb052b769006eb1"}
Nov 03 16:10:44.670140 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:44.669688    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rwpz8" event={"ID":"754fed8d-9c7a-4cee-8806-794ed3330bc0","Type":"ContainerStarted","Data":"d4772de806642000603c5c85a773c96ebac273f46ad5a27176f95945bac50c69"}
Nov 03 16:10:44.670140 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:44.669720    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:44.671581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:44.671551    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qd8hz" event={"ID":"64a4a725-bb12-428a-a3d0-62622448f577","Type":"ContainerStarted","Data":"4f01cdaaf34bdee1dab9ec374f5034b128be31240f1a69cb8be92d26b13049b8"}
Nov 03 16:10:44.673032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:44.673009    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jnx8b" event={"ID":"ef6cea18-3a39-4c4b-8fba-0284ca6f2e98","Type":"ContainerStarted","Data":"09bd4d35f9c8dfe3a933a7b9a53a4239aea9ead8cdf36c1739db0072dab118c9"}
Nov 03 16:10:44.889891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:44.889856    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:10:45.432536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:45.432491    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:10:45.678371 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:45.678331    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s" event={"ID":"d83906ba-ac2d-453a-b42b-3a3d958834ee","Type":"ContainerStarted","Data":"c83aa14547a8ef91073e1226042948eeebad6521a27818d79d0fcf94e4cd46a4"}
Nov 03 16:10:45.678787 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:45.678396    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-8vh2s" event={"ID":"d83906ba-ac2d-453a-b42b-3a3d958834ee","Type":"ContainerStarted","Data":"f76af0c37012bbc7e847719780686be078c17bee018126f04dd45c9a9a43d998"}
Nov 03 16:10:46.836615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:46.836585    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljp5j"
Nov 03 16:10:47.499141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:47.499108    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:47.501379 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:47.501355    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1a241b6-335b-4ee8-add5-07a5e71666e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8szwq\" (UID: \"b1a241b6-335b-4ee8-add5-07a5e71666e5\") " pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:47.720303 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:47.720264    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8szwq"
Nov 03 16:10:48.106252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:48.104330    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8szwq"]
Nov 03 16:10:48.134050 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:48.134012    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a241b6_335b_4ee8_add5_07a5e71666e5.slice/crio-171e56bf60704b5ff9a4fe2650059b0aea06450ad7b374a62c8266832920dbce WatchSource:0}: Error finding container 171e56bf60704b5ff9a4fe2650059b0aea06450ad7b374a62c8266832920dbce: Status 404 returned error can't find the container with id 171e56bf60704b5ff9a4fe2650059b0aea06450ad7b374a62c8266832920dbce
Nov 03 16:10:48.691707 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:48.691671    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8szwq" event={"ID":"b1a241b6-335b-4ee8-add5-07a5e71666e5","Type":"ContainerStarted","Data":"701da59b32b6e0e7c7ceae21ec7527daa035006bb49ec269693e88209e1f7880"}
Nov 03 16:10:48.691876 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:48.691734    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8szwq" event={"ID":"b1a241b6-335b-4ee8-add5-07a5e71666e5","Type":"ContainerStarted","Data":"171e56bf60704b5ff9a4fe2650059b0aea06450ad7b374a62c8266832920dbce"}
Nov 03 16:10:49.701011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:49.700975    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8szwq" event={"ID":"b1a241b6-335b-4ee8-add5-07a5e71666e5","Type":"ContainerStarted","Data":"d9f6accc23cce34eda6bdf8b39799cffe77248759abd69d680483097cc876d6a"}
Nov 03 16:10:51.711519 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:51.711473    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8szwq" event={"ID":"b1a241b6-335b-4ee8-add5-07a5e71666e5","Type":"ContainerStarted","Data":"972705aa30f1b8d1547f4b88f1a54e7a4ac81d9fe40e3ca272ed9d17c00ea314"}
Nov 03 16:10:53.802250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.802212    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"]
Nov 03 16:10:53.806297 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.806274    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"
Nov 03 16:10:53.827938 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.827902    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-xcdx8"]
Nov 03 16:10:53.832145 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.832117    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:53.849268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.849242    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5699b4486-982nb"]
Nov 03 16:10:53.850604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.850579    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f2972ac4-c9e1-45be-ba6d-0b5199d445a7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s557v\" (UID: \"f2972ac4-c9e1-45be-ba6d-0b5199d445a7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"
Nov 03 16:10:53.852761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.852737    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.868876 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.868850    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"]
Nov 03 16:10:53.878236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.874388    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bmf5d\""
Nov 03 16:10:53.878236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.874629    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 16:10:53.886061 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.886031    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 16:10:53.888561 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.888512    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 16:10:53.896693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.896662    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-577d6d7dc8-vdhtd"]
Nov 03 16:10:53.900294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.900261    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.903946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.903922    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 16:10:53.904082 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.904040    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 16:10:53.904082 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.904040    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 16:10:53.949511 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.949478    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xcdx8"]
Nov 03 16:10:53.951559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951523    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-tmp\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.951755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951582    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f1b13c-6e91-4641-b559-17ea289d6862-registry-certificates\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.951755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951682    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhf2h\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-kube-api-access-jhf2h\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.951755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951722    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjpq\" (UniqueName: \"kubernetes.io/projected/e1abd863-3197-490f-af74-0b93cb06cd5a-kube-api-access-lwjpq\") pod \"downloads-6dd789c8b7-xcdx8\" (UID: \"e1abd863-3197-490f-af74-0b93cb06cd5a\") " pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:53.951755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951746    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-serving-cert\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.952005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951802    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-config\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.952005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951873    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f2972ac4-c9e1-45be-ba6d-0b5199d445a7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s557v\" (UID: \"f2972ac4-c9e1-45be-ba6d-0b5199d445a7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"
Nov 03 16:10:53.952005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951940    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-service-ca\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.952005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.951970    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f1b13c-6e91-4641-b559-17ea289d6862-installation-pull-secrets\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.952222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952053    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f1b13c-6e91-4641-b559-17ea289d6862-ca-trust-extracted\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.952222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952102    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1b13c-6e91-4641-b559-17ea289d6862-trusted-ca\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.952222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952177    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-bound-sa-token\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.952381 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952258    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-oauth-config\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.952381 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952294    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6dm\" (UniqueName: \"kubernetes.io/projected/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-kube-api-access-9g6dm\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.952482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952389    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-oauth-serving-cert\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:53.952482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952422    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f1b13c-6e91-4641-b559-17ea289d6862-image-registry-private-configuration\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.952482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952455    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-registry-tls\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:53.952615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.952513    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1abd863-3197-490f-af74-0b93cb06cd5a-tmp\") pod \"downloads-6dd789c8b7-xcdx8\" (UID: \"e1abd863-3197-490f-af74-0b93cb06cd5a\") " pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:53.955181 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.955150    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f2972ac4-c9e1-45be-ba6d-0b5199d445a7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-s557v\" (UID: \"f2972ac4-c9e1-45be-ba6d-0b5199d445a7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"
Nov 03 16:10:53.961693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.961666    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-hkfbh\""
Nov 03 16:10:53.961809 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.961703    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 16:10:53.961852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.961808    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 16:10:53.961852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.961828    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jkpm9\""
Nov 03 16:10:53.996269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:53.996232    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577d6d7dc8-vdhtd"]
Nov 03 16:10:54.007709 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.007676    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"]
Nov 03 16:10:54.053450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053363    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-tmp\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.053450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053404    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f1b13c-6e91-4641-b559-17ea289d6862-registry-certificates\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.053450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053429    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhf2h\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-kube-api-access-jhf2h\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.053450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053449    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjpq\" (UniqueName: \"kubernetes.io/projected/e1abd863-3197-490f-af74-0b93cb06cd5a-kube-api-access-lwjpq\") pod \"downloads-6dd789c8b7-xcdx8\" (UID: \"e1abd863-3197-490f-af74-0b93cb06cd5a\") " pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053465    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-serving-cert\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053516    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-config\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053564    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-service-ca\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053588    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f1b13c-6e91-4641-b559-17ea289d6862-installation-pull-secrets\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053649    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f1b13c-6e91-4641-b559-17ea289d6862-ca-trust-extracted\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053673    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1b13c-6e91-4641-b559-17ea289d6862-trusted-ca\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053866    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-tmp\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.053911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053910    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-bound-sa-token\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.054269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.053980    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-oauth-config\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.054269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054005    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6dm\" (UniqueName: \"kubernetes.io/projected/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-kube-api-access-9g6dm\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.054269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054049    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-oauth-serving-cert\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.054269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054081    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f1b13c-6e91-4641-b559-17ea289d6862-image-registry-private-configuration\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.054269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054120    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-registry-tls\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.054269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054147    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1abd863-3197-490f-af74-0b93cb06cd5a-tmp\") pod \"downloads-6dd789c8b7-xcdx8\" (UID: \"e1abd863-3197-490f-af74-0b93cb06cd5a\") " pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:54.054570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054339    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f1b13c-6e91-4641-b559-17ea289d6862-ca-trust-extracted\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.054570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054452    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f1b13c-6e91-4641-b559-17ea289d6862-registry-certificates\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.054805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054584    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1abd863-3197-490f-af74-0b93cb06cd5a-tmp\") pod \"downloads-6dd789c8b7-xcdx8\" (UID: \"e1abd863-3197-490f-af74-0b93cb06cd5a\") " pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:54.054805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054637    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-service-ca\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.054944 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.054829    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-config\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.055346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.055319    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-oauth-serving-cert\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.055824 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.055798    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1b13c-6e91-4641-b559-17ea289d6862-trusted-ca\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.057629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.057003    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f1b13c-6e91-4641-b559-17ea289d6862-image-registry-private-configuration\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.057629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.057286    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-registry-tls\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.057629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.057291    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-serving-cert\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.057629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.057568    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-oauth-config\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.057629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.057601    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f1b13c-6e91-4641-b559-17ea289d6862-installation-pull-secrets\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.070528 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.070503    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5699b4486-982nb"]
Nov 03 16:10:54.113497 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.113460    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhf2h\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-kube-api-access-jhf2h\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.113828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.113803    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjpq\" (UniqueName: \"kubernetes.io/projected/e1abd863-3197-490f-af74-0b93cb06cd5a-kube-api-access-lwjpq\") pod \"downloads-6dd789c8b7-xcdx8\" (UID: \"e1abd863-3197-490f-af74-0b93cb06cd5a\") " pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:54.113905 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.113886    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6dm\" (UniqueName: \"kubernetes.io/projected/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-kube-api-access-9g6dm\") pod \"console-5699b4486-982nb\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") " pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.114287 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.114267    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f1b13c-6e91-4641-b559-17ea289d6862-bound-sa-token\") pod \"image-registry-577d6d7dc8-vdhtd\" (UID: \"e7f1b13c-6e91-4641-b559-17ea289d6862\") " pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.120232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.120185    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"
Nov 03 16:10:54.145767 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.145730    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:10:54.166650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.166618    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:10:54.212410 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.211562    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.492861 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.492830    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"]
Nov 03 16:10:54.495762 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:54.495732    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2972ac4_c9e1_45be_ba6d_0b5199d445a7.slice/crio-17cd20ed8f8f68cf501a18e8a57390276160dc9924ebea5ca9e23542f9bfd463 WatchSource:0}: Error finding container 17cd20ed8f8f68cf501a18e8a57390276160dc9924ebea5ca9e23542f9bfd463: Status 404 returned error can't find the container with id 17cd20ed8f8f68cf501a18e8a57390276160dc9924ebea5ca9e23542f9bfd463
Nov 03 16:10:54.546752 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:54.546715    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1abd863_3197_490f_af74_0b93cb06cd5a.slice/crio-2d0185fd7bc7bdf5aaaac20a4ddcc4686d273e6925f98c1cea71330f95984913 WatchSource:0}: Error finding container 2d0185fd7bc7bdf5aaaac20a4ddcc4686d273e6925f98c1cea71330f95984913: Status 404 returned error can't find the container with id 2d0185fd7bc7bdf5aaaac20a4ddcc4686d273e6925f98c1cea71330f95984913
Nov 03 16:10:54.547879 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.547861    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xcdx8"]
Nov 03 16:10:54.588299 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.588264    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5699b4486-982nb"]
Nov 03 16:10:54.591742 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:54.591695    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cad5b5_eb89_4b2b_bad2_01c5481fa6f7.slice/crio-93080b8afc3ae6fabd83229c9f9bb9d638e82e7cdb830f67dc9d8bbfc77b4204 WatchSource:0}: Error finding container 93080b8afc3ae6fabd83229c9f9bb9d638e82e7cdb830f67dc9d8bbfc77b4204: Status 404 returned error can't find the container with id 93080b8afc3ae6fabd83229c9f9bb9d638e82e7cdb830f67dc9d8bbfc77b4204
Nov 03 16:10:54.595720 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.595689    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577d6d7dc8-vdhtd"]
Nov 03 16:10:54.598332 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:10:54.598308    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f1b13c_6e91_4641_b559_17ea289d6862.slice/crio-5f89b0c6737ebc6621ac51c8e67e1007b1b67f4202a702d321ea8c25a7758f09 WatchSource:0}: Error finding container 5f89b0c6737ebc6621ac51c8e67e1007b1b67f4202a702d321ea8c25a7758f09: Status 404 returned error can't find the container with id 5f89b0c6737ebc6621ac51c8e67e1007b1b67f4202a702d321ea8c25a7758f09
Nov 03 16:10:54.681128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.681104    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rwpz8"
Nov 03 16:10:54.723967 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.723928    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v" event={"ID":"f2972ac4-c9e1-45be-ba6d-0b5199d445a7","Type":"ContainerStarted","Data":"17cd20ed8f8f68cf501a18e8a57390276160dc9924ebea5ca9e23542f9bfd463"}
Nov 03 16:10:54.725505 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.725474    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd" event={"ID":"e7f1b13c-6e91-4641-b559-17ea289d6862","Type":"ContainerStarted","Data":"46506379d57f9af10a805660332e761f2cb91fbf5d751255e2884251ac1c8062"}
Nov 03 16:10:54.725664 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.725530    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd" event={"ID":"e7f1b13c-6e91-4641-b559-17ea289d6862","Type":"ContainerStarted","Data":"5f89b0c6737ebc6621ac51c8e67e1007b1b67f4202a702d321ea8c25a7758f09"}
Nov 03 16:10:54.725664 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.725563    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:10:54.726759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.726735    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xcdx8" event={"ID":"e1abd863-3197-490f-af74-0b93cb06cd5a","Type":"ContainerStarted","Data":"2d0185fd7bc7bdf5aaaac20a4ddcc4686d273e6925f98c1cea71330f95984913"}
Nov 03 16:10:54.728008 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.727984    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5699b4486-982nb" event={"ID":"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7","Type":"ContainerStarted","Data":"93080b8afc3ae6fabd83229c9f9bb9d638e82e7cdb830f67dc9d8bbfc77b4204"}
Nov 03 16:10:54.907650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:54.907590    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd" podStartSLOduration=1.907573766 podStartE2EDuration="1.907573766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:54.907508213 +0000 UTC m=+16.486812079" watchObservedRunningTime="2026-11-03 16:10:54.907573766 +0000 UTC m=+16.486877613"
Nov 03 16:10:55.436863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:55.436826    5112 patch_prober.go:29] interesting pod/image-registry-687cd8bf85-rh7wb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 16:10:55.437022 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:55.436882    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb" podUID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:10:56.739885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:56.739846    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v" event={"ID":"f2972ac4-c9e1-45be-ba6d-0b5199d445a7","Type":"ContainerStarted","Data":"8d5887b285dc20ef3e168cb497d75b8177ed0ccfc32dc2909523c1348ec8f967"}
Nov 03 16:10:56.740356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:56.739993    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"
Nov 03 16:10:56.745310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:56.745286    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v"
Nov 03 16:10:56.856408 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:56.856347    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v" podStartSLOduration=2.012405926 podStartE2EDuration="3.856328501s" totalImagesPullingTime="1.843922575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:56.8547006 +0000 UTC m=+18.434004456" watchObservedRunningTime="2026-11-03 16:10:56.856328501 +0000 UTC m=+18.435632357"
Nov 03 16:10:58.533983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:58.533945    5112 patch_prober.go:29] interesting pod/image-registry-687cd8bf85-rh7wb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 16:10:58.534490 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:58.534003    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb" podUID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:10:59.014236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:59.014148    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sg6b8"
Nov 03 16:10:59.756476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:10:59.756393    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5699b4486-982nb" event={"ID":"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7","Type":"ContainerStarted","Data":"64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e"}
Nov 03 16:11:03.877468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:03.877438    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:11:04.089264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:04.089175    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5699b4486-982nb" podStartSLOduration=6.214679143 podStartE2EDuration="11.089155101s" totalImagesPullingTime="4.874475958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:10:59.873681565 +0000 UTC m=+21.452985422" watchObservedRunningTime="2026-11-03 16:11:04.089155101 +0000 UTC m=+25.668458956"
Nov 03 16:11:04.167429 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:04.167400    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:11:04.167609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:04.167468    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:11:04.173395 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:04.173371    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:11:04.781265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:04.781236    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:11:04.892921 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:04.892894    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cs98m"
Nov 03 16:11:05.440513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:05.439501    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:11:07.065820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.065785    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c9q49"]
Nov 03 16:11:07.072454 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.072426    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.176750 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.176714    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.176919 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.176762    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.176919 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.176856    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-root\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.176919 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.176902    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-textfile\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.177061 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.176924    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-sys\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.177120 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.177071    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-wtmp\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.177179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.177127    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-metrics-client-ca\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.177179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.177161    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-tls\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.177329 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.177296    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4pc\" (UniqueName: \"kubernetes.io/projected/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-kube-api-access-zh4pc\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.236686 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.236663    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 16:11:07.236849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.236751    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 16:11:07.236849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.236665    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 16:11:07.236849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.236763    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 16:11:07.236849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.236835    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m7glm\""
Nov 03 16:11:07.278476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278447    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278489    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-root\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278519    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-textfile\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278548    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-sys\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278571    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-root\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278589    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-wtmp\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278626    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-sys\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278629    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-metrics-client-ca\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278660    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-tls\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278699    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4pc\" (UniqueName: \"kubernetes.io/projected/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-kube-api-access-zh4pc\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278741    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-wtmp\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.278885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278760    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.279191 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.278914    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-textfile\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.279191 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.279093    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.279318 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.279260    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-metrics-client-ca\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.281356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.281336    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.281566 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.281545    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-node-exporter-tls\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.335732 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.335677    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4pc\" (UniqueName: \"kubernetes.io/projected/2a8de62e-d0ca-4ee7-980a-ae48558a87ee-kube-api-access-zh4pc\") pod \"node-exporter-c9q49\" (UID: \"2a8de62e-d0ca-4ee7-980a-ae48558a87ee\") " pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.391736 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.391711    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9q49"
Nov 03 16:11:07.406351 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:11:07.406324    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8de62e_d0ca_4ee7_980a_ae48558a87ee.slice/crio-b14074928c018f040eb6b155a1574f86a33a09728c492513c9886997b03b0dc0 WatchSource:0}: Error finding container b14074928c018f040eb6b155a1574f86a33a09728c492513c9886997b03b0dc0: Status 404 returned error can't find the container with id b14074928c018f040eb6b155a1574f86a33a09728c492513c9886997b03b0dc0
Nov 03 16:11:07.793442 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:07.793401    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9q49" event={"ID":"2a8de62e-d0ca-4ee7-980a-ae48558a87ee","Type":"ContainerStarted","Data":"b14074928c018f040eb6b155a1574f86a33a09728c492513c9886997b03b0dc0"}
Nov 03 16:11:07.888225 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:11:07.888114    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:11:08.263858 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:08.263824    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5699b4486-982nb"]
Nov 03 16:11:09.804259 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:09.804222    5112 generic.go:356] "Generic (PLEG): container finished" podID="2a8de62e-d0ca-4ee7-980a-ae48558a87ee" containerID="316e30291a243a35d947bb10ab3e60dd31074e47ae2b09e5749da3de5ed431e8" exitCode=0
Nov 03 16:11:09.804719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:09.804291    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9q49" event={"ID":"2a8de62e-d0ca-4ee7-980a-ae48558a87ee","Type":"ContainerDied","Data":"316e30291a243a35d947bb10ab3e60dd31074e47ae2b09e5749da3de5ed431e8"}
Nov 03 16:11:10.811577 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:10.811534    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9q49" event={"ID":"2a8de62e-d0ca-4ee7-980a-ae48558a87ee","Type":"ContainerStarted","Data":"307ce2b274c87ac6a7fe70387e7a1e1b1c2e88041f8a34a80868a1096090ba68"}
Nov 03 16:11:10.812042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:10.811593    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9q49" event={"ID":"2a8de62e-d0ca-4ee7-980a-ae48558a87ee","Type":"ContainerStarted","Data":"1640454fbf3b3b22c59f4f9bcd55e224d45af8b07a6c485251802cbb50773292"}
Nov 03 16:11:10.927436 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:10.927380    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c9q49" podStartSLOduration=2.284432436 podStartE2EDuration="3.927362632s" totalImagesPullingTime="1.567025016s" totalInitContainerRuntime="75.90518ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:10.926611756 +0000 UTC m=+32.505915646" watchObservedRunningTime="2026-11-03 16:11:10.927362632 +0000 UTC m=+32.506666518"
Nov 03 16:11:12.822916 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:12.822852    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xcdx8" event={"ID":"e1abd863-3197-490f-af74-0b93cb06cd5a","Type":"ContainerStarted","Data":"bfc6f0f36add5f81487f6f42bc686accde7e65684e38f75cc7eaa7eab677101f"}
Nov 03 16:11:12.823418 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:12.822943    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:11:12.835770 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:12.835740    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-xcdx8"
Nov 03 16:11:13.029860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:13.029797    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-xcdx8" podStartSLOduration=2.727698829 podStartE2EDuration="20.029782515s" totalImagesPullingTime="17.302083686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:11:12.981482668 +0000 UTC m=+34.560786559" watchObservedRunningTime="2026-11-03 16:11:13.029782515 +0000 UTC m=+34.609086367"
Nov 03 16:11:15.738488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:15.738456    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-577d6d7dc8-vdhtd"
Nov 03 16:11:15.953536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:15.953496    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687cd8bf85-rh7wb"]
Nov 03 16:11:18.900699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:18.900628    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h" podUID="9cf495d7-831a-4266-a508-c12d9968971d" containerName="registry" containerID="cri-o://f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b" gracePeriod=30
Nov 03 16:11:19.214700 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.214672    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:11:19.387432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387399    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.387693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387459    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.387693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387523    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.387693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387571    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.387693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387635    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.387693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387670    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.387990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387741    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.387990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.387772    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\") pod \"9cf495d7-831a-4266-a508-c12d9968971d\" (UID: \"9cf495d7-831a-4266-a508-c12d9968971d\") "
Nov 03 16:11:19.388928 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.388874    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:11:19.389356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.389176    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:11:19.390536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.390511    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:19.393408 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.393376    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:11:19.397341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.397316    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:19.400648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.400625    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:19.400790 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.400736    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:11:19.402274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.402244    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl" pod "9cf495d7-831a-4266-a508-c12d9968971d" (UID: "9cf495d7-831a-4266-a508-c12d9968971d"). InnerVolumeSpecName "kube-api-access-jsqfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:19.489154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489090    5112 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-image-registry-private-configuration\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.489154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489125    5112 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-registry-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.489154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489142    5112 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cf495d7-831a-4266-a508-c12d9968971d-ca-trust-extracted\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.489387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489157    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsqfl\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-kube-api-access-jsqfl\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.489387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489172    5112 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cf495d7-831a-4266-a508-c12d9968971d-installation-pull-secrets\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.489387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489186    5112 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cf495d7-831a-4266-a508-c12d9968971d-bound-sa-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.489387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489216    5112 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-registry-certificates\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.489387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.489228    5112 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cf495d7-831a-4266-a508-c12d9968971d-trusted-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:19.856573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.856498    5112 generic.go:356] "Generic (PLEG): container finished" podID="9cf495d7-831a-4266-a508-c12d9968971d" containerID="f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b" exitCode=0
Nov 03 16:11:19.856573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.856563    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h" event={"ID":"9cf495d7-831a-4266-a508-c12d9968971d","Type":"ContainerDied","Data":"f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b"}
Nov 03 16:11:19.856902 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.856569    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"
Nov 03 16:11:19.856902 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.856614    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h" event={"ID":"9cf495d7-831a-4266-a508-c12d9968971d","Type":"ContainerDied","Data":"de057a33fdc0274d400863a00be548874e51d8ef4232b52adeceb1e79fd17777"}
Nov 03 16:11:19.856902 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.856640    5112 scope.go:122] "RemoveContainer" containerID="f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b"
Nov 03 16:11:19.867680 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.867653    5112 scope.go:122] "RemoveContainer" containerID="f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b"
Nov 03 16:11:19.868021 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:11:19.867991    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b\": container with ID starting with f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b not found: ID does not exist" containerID="f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b"
Nov 03 16:11:19.868118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:19.868033    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b"} err="failed to get container status \"f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b\": rpc error: code = NotFound desc = could not find container \"f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b\": container with ID starting with f5ef169c237dbeaffd8a3e63f3312cef6a854a332ca1b10a6f79b5f5374a734b not found: ID does not exist"
Nov 03 16:11:20.030026 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:20.030001    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"]
Nov 03 16:11:20.076973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:20.076946    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f4bdbf4dd-bqs7h"]
Nov 03 16:11:20.550048 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:20.550013    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf495d7-831a-4266-a508-c12d9968971d" path="/var/lib/kubelet/pods/9cf495d7-831a-4266-a508-c12d9968971d/volumes"
Nov 03 16:11:33.286928 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.286874    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5699b4486-982nb" podUID="11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" containerName="console" containerID="cri-o://64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e" gracePeriod=15
Nov 03 16:11:33.569618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.569593    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:11:33.590064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590037    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-serving-cert\") pod \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") "
Nov 03 16:11:33.590180 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590071    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-config\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-config\") pod \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") "
Nov 03 16:11:33.590180 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590106    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-kube-api-access-9g6dm\" (UniqueName: \"kubernetes.io/projected/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-kube-api-access-9g6dm\") pod \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") "
Nov 03 16:11:33.590180 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590162    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-oauth-config\") pod \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") "
Nov 03 16:11:33.590364 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590222    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-tmp\") pod \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") "
Nov 03 16:11:33.590364 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590260    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-service-ca\") pod \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") "
Nov 03 16:11:33.590364 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590294    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-oauth-serving-cert\") pod \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\" (UID: \"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7\") "
Nov 03 16:11:33.591007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590976    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-oauth-serving-cert" pod "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" (UID: "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:11:33.592226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.590460    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-config" pod "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" (UID: "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:11:33.592226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.591525    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-tmp" pod "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" (UID: "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:33.592226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.591589    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-service-ca" pod "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" (UID: "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:11:33.593351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.593317    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-serving-cert" pod "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" (UID: "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:11:33.596911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.596888    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-oauth-config" pod "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" (UID: "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:11:33.597699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.597677    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-kube-api-access-9g6dm" pod "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" (UID: "11cad5b5-eb89-4b2b-bad2-01c5481fa6f7"). InnerVolumeSpecName "kube-api-access-9g6dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:33.691810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.691785    5112 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-tmp\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:33.691810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.691808    5112 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:33.691936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.691819    5112 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-oauth-serving-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:33.691936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.691828    5112 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-serving-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:33.691936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.691837    5112 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-config\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:33.691936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.691846    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6dm\" (UniqueName: \"kubernetes.io/projected/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-kube-api-access-9g6dm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:33.691936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.691855    5112 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7-console-oauth-config\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:33.913085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.913057    5112 generic.go:356] "Generic (PLEG): container finished" podID="11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" containerID="64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e" exitCode=0
Nov 03 16:11:33.913257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.913105    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5699b4486-982nb" event={"ID":"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7","Type":"ContainerDied","Data":"64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e"}
Nov 03 16:11:33.913257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.913124    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5699b4486-982nb"
Nov 03 16:11:33.913257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.913167    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5699b4486-982nb" event={"ID":"11cad5b5-eb89-4b2b-bad2-01c5481fa6f7","Type":"ContainerDied","Data":"93080b8afc3ae6fabd83229c9f9bb9d638e82e7cdb830f67dc9d8bbfc77b4204"}
Nov 03 16:11:33.913257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.913187    5112 scope.go:122] "RemoveContainer" containerID="64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e"
Nov 03 16:11:33.921621 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.921596    5112 scope.go:122] "RemoveContainer" containerID="64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e"
Nov 03 16:11:33.921876 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:11:33.921856    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e\": container with ID starting with 64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e not found: ID does not exist" containerID="64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e"
Nov 03 16:11:33.921937 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:33.921884    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e"} err="failed to get container status \"64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e\": rpc error: code = NotFound desc = could not find container \"64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e\": container with ID starting with 64f75b8f14ffd7bcc941d58729f01f54a9819ff99b3b63998fa35821e304d13e not found: ID does not exist"
Nov 03 16:11:34.071853 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:34.071824    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5699b4486-982nb"]
Nov 03 16:11:34.119391 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:34.119365    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5699b4486-982nb"]
Nov 03 16:11:34.548310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:34.548280    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" path="/var/lib/kubelet/pods/11cad5b5-eb89-4b2b-bad2-01c5481fa6f7/volumes"
Nov 03 16:11:37.881986 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:11:37.881948    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:11:40.978510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:40.978445    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb" podUID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" containerName="registry" containerID="cri-o://90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649" gracePeriod=30
Nov 03 16:11:41.271544 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.271513    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:11:41.357316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357277    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.357510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357326    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.357510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357368    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.357510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357387    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.357510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357411    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.357510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357439    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.357510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357476    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.357846 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357520    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\") pod \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\" (UID: \"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573\") "
Nov 03 16:11:41.358001 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.357966    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:11:41.358080 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.358056    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:11:41.360008 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.359942    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:11:41.360135 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.360068    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:11:41.360214 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.360152    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:41.360299 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.360279    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:41.360446 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.360423    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "kube-api-access-lrvcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:11:41.364630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.364599    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted" pod "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" (UID: "9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:11:41.458822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458788    5112 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.458822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458821    5112 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-image-registry-private-configuration\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.458822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458833    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrvcd\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-kube-api-access-lrvcd\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.459069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458844    5112 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-trusted-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.459069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458853    5112 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-bound-sa-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.459069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458862    5112 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-ca-trust-extracted\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.459069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458870    5112 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-installation-pull-secrets\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.459069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.458880    5112 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573-registry-certificates\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:11:41.948316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.948275    5112 generic.go:356] "Generic (PLEG): container finished" podID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" containerID="90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649" exitCode=0
Nov 03 16:11:41.948316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.948320    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb"
Nov 03 16:11:41.948536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.948324    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb" event={"ID":"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573","Type":"ContainerDied","Data":"90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649"}
Nov 03 16:11:41.948536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.948373    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687cd8bf85-rh7wb" event={"ID":"9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573","Type":"ContainerDied","Data":"f90782f3bc4e74abe17c3c536942ea0fcc6dff70336fe116741694b3ff1a1f06"}
Nov 03 16:11:41.948536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.948395    5112 scope.go:122] "RemoveContainer" containerID="90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649"
Nov 03 16:11:41.957393 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.957374    5112 scope.go:122] "RemoveContainer" containerID="90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649"
Nov 03 16:11:41.957686 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:11:41.957658    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649\": container with ID starting with 90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649 not found: ID does not exist" containerID="90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649"
Nov 03 16:11:41.957724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:41.957694    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649"} err="failed to get container status \"90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649\": rpc error: code = NotFound desc = could not find container \"90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649\": container with ID starting with 90157c9ec9c495042d3b89fffb2a86b6218d81e1a88184bc19c423004ad5b649 not found: ID does not exist"
Nov 03 16:11:42.155264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:42.155231    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687cd8bf85-rh7wb"]
Nov 03 16:11:42.202593 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:42.202497    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-687cd8bf85-rh7wb"]
Nov 03 16:11:42.547010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:42.546926    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" path="/var/lib/kubelet/pods/9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573/volumes"
Nov 03 16:11:59.018858 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:59.018826    5112 generic.go:356] "Generic (PLEG): container finished" podID="f7cd889c-edce-4208-9b4f-005d200d566b" containerID="23c35647f028e8675616f632e43811e95bed9a5501db28165a3b3d8e91c5b87d" exitCode=0
Nov 03 16:11:59.019246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:59.018896    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh" event={"ID":"f7cd889c-edce-4208-9b4f-005d200d566b","Type":"ContainerDied","Data":"23c35647f028e8675616f632e43811e95bed9a5501db28165a3b3d8e91c5b87d"}
Nov 03 16:11:59.019246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:11:59.019186    5112 scope.go:122] "RemoveContainer" containerID="23c35647f028e8675616f632e43811e95bed9a5501db28165a3b3d8e91c5b87d"
Nov 03 16:12:00.026563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:00.026525    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p6mlh" event={"ID":"f7cd889c-edce-4208-9b4f-005d200d566b","Type":"ContainerStarted","Data":"de326a43fdbb1e0f436436e4073c93478a88b3115bc5f6f17036e007e8b5e991"}
Nov 03 16:12:03.039348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:03.039317    5112 generic.go:356] "Generic (PLEG): container finished" podID="7351e46a-1bd0-430b-aec5-1f3111e80370" containerID="6f6b03577588e47dcf604ce92d13f39c41ca03d2f648fd8d0cff7f94c6508e2f" exitCode=0
Nov 03 16:12:03.039843 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:03.039394    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l75v9" event={"ID":"7351e46a-1bd0-430b-aec5-1f3111e80370","Type":"ContainerDied","Data":"6f6b03577588e47dcf604ce92d13f39c41ca03d2f648fd8d0cff7f94c6508e2f"}
Nov 03 16:12:03.039843 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:03.039796    5112 scope.go:122] "RemoveContainer" containerID="6f6b03577588e47dcf604ce92d13f39c41ca03d2f648fd8d0cff7f94c6508e2f"
Nov 03 16:12:04.044928 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.044897    5112 generic.go:356] "Generic (PLEG): container finished" podID="43e7bf8b-0e8b-46e5-a55a-12dd965a41d0" containerID="20a0b4c77827b92d95adf0b7981eaf0959c245e637930dbdc85590f649aeeba3" exitCode=0
Nov 03 16:12:04.045392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.044968    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b" event={"ID":"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0","Type":"ContainerDied","Data":"20a0b4c77827b92d95adf0b7981eaf0959c245e637930dbdc85590f649aeeba3"}
Nov 03 16:12:04.045392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.045306    5112 scope.go:122] "RemoveContainer" containerID="20a0b4c77827b92d95adf0b7981eaf0959c245e637930dbdc85590f649aeeba3"
Nov 03 16:12:04.046936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.046915    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-l75v9" event={"ID":"7351e46a-1bd0-430b-aec5-1f3111e80370","Type":"ContainerStarted","Data":"279d8fd25ddf23d7430e6b23ad2c238fe6dbda280cf247994185f4ddb280c317"}
Nov 03 16:12:04.095438 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.095406    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xxtp9-hcmjv"]
Nov 03 16:12:04.095916 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.095867    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv" podUID="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" containerName="insights-gathering" containerID="cri-o://3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32" gracePeriod=30
Nov 03 16:12:04.361742 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.361718    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:12:04.446588 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.446546    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\") pod \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") "
Nov 03 16:12:04.446760 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.446629    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\") pod \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") "
Nov 03 16:12:04.446760 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.446672    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\" (UniqueName: \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\") pod \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") "
Nov 03 16:12:04.446760 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.446736    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\") pod \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\" (UID: \"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e\") "
Nov 03 16:12:04.447125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.446996    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle" pod "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" (UID: "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:12:04.447213 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.447001    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path" pod "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" (UID: "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:12:04.447213 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.447139    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle" pod "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" (UID: "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:12:04.449091 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.449064    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5" pod "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" (UID: "5ad56a4f-f2b5-4dae-8bdb-36a008d1645e"). InnerVolumeSpecName "kube-api-access-tsjk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:12:04.547314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.547282    5112 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-trusted-ca-bundle\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:12:04.547314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.547310    5112 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-archives-path\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:12:04.547486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.547323    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjk5\" (UniqueName: \"kubernetes.io/projected/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-kube-api-access-tsjk5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:12:04.547486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:04.547338    5112 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e-service-ca-bundle\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:12:05.051862 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.051822    5112 generic.go:356] "Generic (PLEG): container finished" podID="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" containerID="3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32" exitCode=2
Nov 03 16:12:05.052324 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.051868    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv"
Nov 03 16:12:05.052324 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.051902    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv" event={"ID":"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e","Type":"ContainerDied","Data":"3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32"}
Nov 03 16:12:05.052324 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.051964    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xxtp9-hcmjv" event={"ID":"5ad56a4f-f2b5-4dae-8bdb-36a008d1645e","Type":"ContainerDied","Data":"4bcc3d54aa3ef08bc04e37ada1a9ee3deac6f8367ec85e8cb8663804ee731779"}
Nov 03 16:12:05.052324 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.051993    5112 scope.go:122] "RemoveContainer" containerID="3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32"
Nov 03 16:12:05.054742 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.054700    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-vxg4b" event={"ID":"43e7bf8b-0e8b-46e5-a55a-12dd965a41d0","Type":"ContainerStarted","Data":"dbba846d8e1c261588b3f81346811a3d23ff62565ad82666a0fd365f542b1330"}
Nov 03 16:12:05.060432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.060404    5112 scope.go:122] "RemoveContainer" containerID="3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32"
Nov 03 16:12:05.060668 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:12:05.060650    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32\": container with ID starting with 3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32 not found: ID does not exist" containerID="3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32"
Nov 03 16:12:05.060721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.060675    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32"} err="failed to get container status \"3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32\": rpc error: code = NotFound desc = could not find container \"3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32\": container with ID starting with 3b2e12dbd3013d90c2f2c7c3d4993637cd5d9a2e27a1b285a3039d70a4c13b32 not found: ID does not exist"
Nov 03 16:12:05.211119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.211087    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xxtp9-hcmjv"]
Nov 03 16:12:05.257848 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:05.257818    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-xxtp9-hcmjv"]
Nov 03 16:12:06.547881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:12:06.547851    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" path="/var/lib/kubelet/pods/5ad56a4f-f2b5-4dae-8bdb-36a008d1645e/volumes"
Nov 03 16:12:07.880856 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:12:07.880820    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:12:37.880947 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:12:37.880908    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:13:07.880764 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:13:07.880728    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:13:37.882128 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:13:37.882086    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:14:07.881670 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:14:07.881583    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:14:37.034852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.034825    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"]
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035126    5112 cpu_manager.go:414] "containerMap: removing container" podUID="11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" containerName="console"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035136    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" containerName="console"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035154    5112 cpu_manager.go:414] "containerMap: removing container" podUID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" containerName="registry"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035159    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" containerName="registry"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035167    5112 cpu_manager.go:414] "containerMap: removing container" podUID="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" containerName="insights-gathering"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035171    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" containerName="insights-gathering"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035186    5112 cpu_manager.go:414] "containerMap: removing container" podUID="9cf495d7-831a-4266-a508-c12d9968971d" containerName="registry"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035209    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf495d7-831a-4266-a508-c12d9968971d" containerName="registry"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035277    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1ab42f-45fe-4c02-89b8-3fc4aa9fb573" containerName="registry"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035285    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad56a4f-f2b5-4dae-8bdb-36a008d1645e" containerName="insights-gathering"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035293    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf495d7-831a-4266-a508-c12d9968971d" containerName="registry"
Nov 03 16:14:37.035316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.035301    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cad5b5-eb89-4b2b-bad2-01c5481fa6f7" containerName="console"
Nov 03 16:14:37.038581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.038566    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.090571 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.090548    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 16:14:37.136209 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.136162    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 16:14:37.136541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.136519    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-g4gf7\""
Nov 03 16:14:37.138746 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.138722    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"]
Nov 03 16:14:37.162469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.162448    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.162570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.162506    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.162628 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.162608    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/5342e8ae-34c1-4f6a-b671-24546d54380e-kube-api-access-8mb4d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.263563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.263536    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.263677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.263571    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/5342e8ae-34c1-4f6a-b671-24546d54380e-kube-api-access-8mb4d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.263677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.263596    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.263944 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.263922    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.264014 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.263974    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.320095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.320038    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/5342e8ae-34c1-4f6a-b671-24546d54380e-kube-api-access-8mb4d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.353973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.353942    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:14:37.718926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:37.718899    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"]
Nov 03 16:14:37.721647 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:14:37.721621    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5342e8ae_34c1_4f6a_b671_24546d54380e.slice/crio-aea3257ba37455736740a642b305233756d84fdeb2f30a77d4b272aac6e8c747 WatchSource:0}: Error finding container aea3257ba37455736740a642b305233756d84fdeb2f30a77d4b272aac6e8c747: Status 404 returned error can't find the container with id aea3257ba37455736740a642b305233756d84fdeb2f30a77d4b272aac6e8c747
Nov 03 16:14:37.880124 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:14:37.880094    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:14:38.646868 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.646837    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84" event={"ID":"5342e8ae-34c1-4f6a-b671-24546d54380e","Type":"ContainerStarted","Data":"aea3257ba37455736740a642b305233756d84fdeb2f30a77d4b272aac6e8c747"}
Nov 03 16:14:38.891567 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.891532    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"]
Nov 03 16:14:38.895449 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.895422    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:38.975059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.975027    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2t4\" (UniqueName: \"kubernetes.io/projected/f6e8d4ca-17c7-4c23-bc88-934fa6824168-kube-api-access-8r2t4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:38.975246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.975107    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:38.975246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.975180    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:38.993375 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.993346    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-h5mw9\""
Nov 03 16:14:38.994884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:38.994853    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"]
Nov 03 16:14:39.076520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.076487    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:39.076520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.076531    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:39.076768 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.076605    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2t4\" (UniqueName: \"kubernetes.io/projected/f6e8d4ca-17c7-4c23-bc88-934fa6824168-kube-api-access-8r2t4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:39.077217 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.077178    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:39.077330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.077268    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:39.133989 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.133953    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2t4\" (UniqueName: \"kubernetes.io/projected/f6e8d4ca-17c7-4c23-bc88-934fa6824168-kube-api-access-8r2t4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:39.214397 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.214317    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:14:39.593406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.593337    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"]
Nov 03 16:14:39.623779 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:14:39.623755    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e8d4ca_17c7_4c23_bc88_934fa6824168.slice/crio-7931b682a7d71e2731e5604f189f1c6e3faee4c874f2b3b493f5bd0b0eaca9ef WatchSource:0}: Error finding container 7931b682a7d71e2731e5604f189f1c6e3faee4c874f2b3b493f5bd0b0eaca9ef: Status 404 returned error can't find the container with id 7931b682a7d71e2731e5604f189f1c6e3faee4c874f2b3b493f5bd0b0eaca9ef
Nov 03 16:14:39.652170 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:39.652141    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst" event={"ID":"f6e8d4ca-17c7-4c23-bc88-934fa6824168","Type":"ContainerStarted","Data":"7931b682a7d71e2731e5604f189f1c6e3faee4c874f2b3b493f5bd0b0eaca9ef"}
Nov 03 16:14:40.288469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.288432    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"]
Nov 03 16:14:40.293144 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.293125    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.348932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.348869    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"]
Nov 03 16:14:40.349077 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.349039    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mjwsb\""
Nov 03 16:14:40.388432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.388398    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.388592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.388443    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.388592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.388463    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqjv9\" (UniqueName: \"kubernetes.io/projected/1489bd54-2e77-417f-a157-f83840cd1e74-kube-api-access-rqjv9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.489893 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.489861    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.490057 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.489911    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.490057 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.489987    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqjv9\" (UniqueName: \"kubernetes.io/projected/1489bd54-2e77-417f-a157-f83840cd1e74-kube-api-access-rqjv9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.490277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.490260    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.490369 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.490316    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.547437 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.547354    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqjv9\" (UniqueName: \"kubernetes.io/projected/1489bd54-2e77-417f-a157-f83840cd1e74-kube-api-access-rqjv9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.611212 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.611167    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:14:40.988854 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:40.988817    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"]
Nov 03 16:14:40.993173 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:14:40.993140    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1489bd54_2e77_417f_a157_f83840cd1e74.slice/crio-853bffd37fb9f5ca8e5097554b4474a8198f1ac605d5747a8746adb1a596e887 WatchSource:0}: Error finding container 853bffd37fb9f5ca8e5097554b4474a8198f1ac605d5747a8746adb1a596e887: Status 404 returned error can't find the container with id 853bffd37fb9f5ca8e5097554b4474a8198f1ac605d5747a8746adb1a596e887
Nov 03 16:14:41.665647 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:41.665611    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" event={"ID":"1489bd54-2e77-417f-a157-f83840cd1e74","Type":"ContainerStarted","Data":"853bffd37fb9f5ca8e5097554b4474a8198f1ac605d5747a8746adb1a596e887"}
Nov 03 16:14:43.056708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.056672    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"]
Nov 03 16:14:43.060738 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.060711    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.112963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.112920    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.112982    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.113019    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.113082    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.113107    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.113131    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tmq\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-kube-api-access-97tmq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.113149    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.113180    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.113325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.113209    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.132799 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.132690    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mw47n\""
Nov 03 16:14:43.132799 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.132690    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 16:14:43.162848 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.162818    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"]
Nov 03 16:14:43.214744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214702    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.214927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214755    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.214927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214790    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.214927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214846    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.214927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214872    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.214927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214904    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tmq\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-kube-api-access-97tmq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.215221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214929    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.215221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.214979    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.215221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.215006    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.215600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.215574    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.215677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.215662    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.216245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.216217    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.216341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.216266    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.216341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.216270    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.218396 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.218370    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.220022 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.220001    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.273348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.273291    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tmq\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-kube-api-access-97tmq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.273489 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.273434    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sbbf7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.376421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.376338    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:43.801394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:43.800987    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"]
Nov 03 16:14:43.803340 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:14:43.803298    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725cb813_4d0c_4922_b79c_ec7c2624d5a7.slice/crio-c9fe46ec37c3cb135430d4016f4772ac2d9c815ff1f3b9f69862c89df55928ca WatchSource:0}: Error finding container c9fe46ec37c3cb135430d4016f4772ac2d9c815ff1f3b9f69862c89df55928ca: Status 404 returned error can't find the container with id c9fe46ec37c3cb135430d4016f4772ac2d9c815ff1f3b9f69862c89df55928ca
Nov 03 16:14:44.683010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:44.682970    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" event={"ID":"725cb813-4d0c-4922-b79c-ec7c2624d5a7","Type":"ContainerStarted","Data":"c9fe46ec37c3cb135430d4016f4772ac2d9c815ff1f3b9f69862c89df55928ca"}
Nov 03 16:14:48.705557 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:48.705514    5112 generic.go:356] "Generic (PLEG): container finished" podID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerID="18d5f3ce98fe80f48267db62cceea93991fd708ae5540a90c92bbf062c30a0ae" exitCode=0
Nov 03 16:14:48.705961 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:48.705599    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst" event={"ID":"f6e8d4ca-17c7-4c23-bc88-934fa6824168","Type":"ContainerDied","Data":"18d5f3ce98fe80f48267db62cceea93991fd708ae5540a90c92bbf062c30a0ae"}
Nov 03 16:14:49.712685 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:49.712646    5112 generic.go:356] "Generic (PLEG): container finished" podID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerID="af80688e40f6acaf02b593f33a601be2162b9637784ae27e27162fdc40997717" exitCode=0
Nov 03 16:14:49.713170 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:49.712782    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84" event={"ID":"5342e8ae-34c1-4f6a-b671-24546d54380e","Type":"ContainerDied","Data":"af80688e40f6acaf02b593f33a601be2162b9637784ae27e27162fdc40997717"}
Nov 03 16:14:50.633424 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:50.633384    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:14:50.633565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:50.633473    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:14:50.633565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:50.633515    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:14:50.722811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:50.722781    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" event={"ID":"725cb813-4d0c-4922-b79c-ec7c2624d5a7","Type":"ContainerStarted","Data":"50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51"}
Nov 03 16:14:50.859433 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:50.859376    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" podStartSLOduration=1.032254237 podStartE2EDuration="7.859363065s" totalImagesPullingTime="6.827108828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:50.859276393 +0000 UTC m=+252.438580265" watchObservedRunningTime="2026-11-03 16:14:50.859363065 +0000 UTC m=+252.438666916"
Nov 03 16:14:51.377395 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:51.377368    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:51.381986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:51.381966    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:51.731292 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:51.731256    5112 generic.go:356] "Generic (PLEG): container finished" podID="1489bd54-2e77-417f-a157-f83840cd1e74" containerID="df48ae2ea8fa150d2425f61f86855d517d45ab81901f91f02eeddefb88ad89f8" exitCode=0
Nov 03 16:14:51.731792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:51.731332    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" event={"ID":"1489bd54-2e77-417f-a157-f83840cd1e74","Type":"ContainerDied","Data":"df48ae2ea8fa150d2425f61f86855d517d45ab81901f91f02eeddefb88ad89f8"}
Nov 03 16:14:51.731792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:51.731699    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:51.732965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:51.732848    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:14:52.737469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:52.737444    5112 generic.go:356] "Generic (PLEG): container finished" podID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerID="1085970babb0b429c58129a65bf425ba011cdb9589d5c7c77342090796eb734c" exitCode=0
Nov 03 16:14:52.737848 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:52.737529    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84" event={"ID":"5342e8ae-34c1-4f6a-b671-24546d54380e","Type":"ContainerDied","Data":"1085970babb0b429c58129a65bf425ba011cdb9589d5c7c77342090796eb734c"}
Nov 03 16:14:52.739487 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:52.739462    5112 generic.go:356] "Generic (PLEG): container finished" podID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerID="eae17ef13adf66a6476035c9c0b981ef978a1c4da04b14198ef01b60769220ab" exitCode=0
Nov 03 16:14:52.739601 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:52.739545    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst" event={"ID":"f6e8d4ca-17c7-4c23-bc88-934fa6824168","Type":"ContainerDied","Data":"eae17ef13adf66a6476035c9c0b981ef978a1c4da04b14198ef01b60769220ab"}
Nov 03 16:14:52.741526 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:52.741508    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" event={"ID":"1489bd54-2e77-417f-a157-f83840cd1e74","Type":"ContainerStarted","Data":"dccca317df5ed69e4abb0ccc9f6a1b491c69d5b8dd8bb7c77bf921bb8c847bda"}
Nov 03 16:14:53.751068 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:53.751024    5112 generic.go:356] "Generic (PLEG): container finished" podID="1489bd54-2e77-417f-a157-f83840cd1e74" containerID="dccca317df5ed69e4abb0ccc9f6a1b491c69d5b8dd8bb7c77bf921bb8c847bda" exitCode=0
Nov 03 16:14:53.751713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:14:53.751127    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" event={"ID":"1489bd54-2e77-417f-a157-f83840cd1e74","Type":"ContainerDied","Data":"dccca317df5ed69e4abb0ccc9f6a1b491c69d5b8dd8bb7c77bf921bb8c847bda"}
Nov 03 16:15:01.797324 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:01.797282    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" event={"ID":"1489bd54-2e77-417f-a157-f83840cd1e74","Type":"ContainerStarted","Data":"6e91f223323518ae231e3386e3b32ca47cf812917b65de38a94d3cfa471e64f5"}
Nov 03 16:15:02.804922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:02.804883    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:15:02.805767 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:02.805735    5112 generic.go:356] "Generic (PLEG): container finished" podID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerID="cca8872ec1d6b053b177003f8048b5bb587992b48c86b8d0e5cbe7c4dba439e1" exitCode=1
Nov 03 16:15:02.805905 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:02.805822    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst" event={"ID":"f6e8d4ca-17c7-4c23-bc88-934fa6824168","Type":"ContainerDied","Data":"cca8872ec1d6b053b177003f8048b5bb587992b48c86b8d0e5cbe7c4dba439e1"}
Nov 03 16:15:02.808246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:02.808212    5112 generic.go:356] "Generic (PLEG): container finished" podID="1489bd54-2e77-417f-a157-f83840cd1e74" containerID="6e91f223323518ae231e3386e3b32ca47cf812917b65de38a94d3cfa471e64f5" exitCode=0
Nov 03 16:15:02.808377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:02.808292    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" event={"ID":"1489bd54-2e77-417f-a157-f83840cd1e74","Type":"ContainerDied","Data":"6e91f223323518ae231e3386e3b32ca47cf812917b65de38a94d3cfa471e64f5"}
Nov 03 16:15:02.920845 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:02.920790    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" podStartSLOduration=4.115449002 podStartE2EDuration="22.920769815s" totalImagesPullingTime="18.805320813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:01.943928399 +0000 UTC m=+263.523232272" watchObservedRunningTime="2026-11-03 16:15:02.920769815 +0000 UTC m=+264.500073672"
Nov 03 16:15:03.814109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:03.814077    5112 generic.go:356] "Generic (PLEG): container finished" podID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerID="49824524af4a379f14dce8b9237f9efb1b23718c15044f1f81df34d90170c975" exitCode=0
Nov 03 16:15:03.814528 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:03.814160    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84" event={"ID":"5342e8ae-34c1-4f6a-b671-24546d54380e","Type":"ContainerDied","Data":"49824524af4a379f14dce8b9237f9efb1b23718c15044f1f81df34d90170c975"}
Nov 03 16:15:03.942911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:03.942887    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:15:03.945961 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:03.945936    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:15:03.963494 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:03.963468    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:15:04.111258 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.111175    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e8d4ca-17c7-4c23-bc88-934fa6824168-kube-api-access-8r2t4\" (UniqueName: \"kubernetes.io/projected/f6e8d4ca-17c7-4c23-bc88-934fa6824168-kube-api-access-8r2t4\") pod \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") "
Nov 03 16:15:04.111258 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.111251    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-bundle\") pod \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") "
Nov 03 16:15:04.111488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.111273    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-util\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-util\") pod \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\" (UID: \"f6e8d4ca-17c7-4c23-bc88-934fa6824168\") "
Nov 03 16:15:04.111488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.111303    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-bundle\") pod \"1489bd54-2e77-417f-a157-f83840cd1e74\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") "
Nov 03 16:15:04.111488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.111352    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-util\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-util\") pod \"1489bd54-2e77-417f-a157-f83840cd1e74\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") "
Nov 03 16:15:04.111488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.111391    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1489bd54-2e77-417f-a157-f83840cd1e74-kube-api-access-rqjv9\" (UniqueName: \"kubernetes.io/projected/1489bd54-2e77-417f-a157-f83840cd1e74-kube-api-access-rqjv9\") pod \"1489bd54-2e77-417f-a157-f83840cd1e74\" (UID: \"1489bd54-2e77-417f-a157-f83840cd1e74\") "
Nov 03 16:15:04.111768 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.111706    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-bundle" pod "f6e8d4ca-17c7-4c23-bc88-934fa6824168" (UID: "f6e8d4ca-17c7-4c23-bc88-934fa6824168"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:15:04.112142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.112106    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-bundle" pod "1489bd54-2e77-417f-a157-f83840cd1e74" (UID: "1489bd54-2e77-417f-a157-f83840cd1e74"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:15:04.113718 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.113690    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1489bd54-2e77-417f-a157-f83840cd1e74-kube-api-access-rqjv9" pod "1489bd54-2e77-417f-a157-f83840cd1e74" (UID: "1489bd54-2e77-417f-a157-f83840cd1e74"). InnerVolumeSpecName "kube-api-access-rqjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:15:04.113808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.113734    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e8d4ca-17c7-4c23-bc88-934fa6824168-kube-api-access-8r2t4" pod "f6e8d4ca-17c7-4c23-bc88-934fa6824168" (UID: "f6e8d4ca-17c7-4c23-bc88-934fa6824168"). InnerVolumeSpecName "kube-api-access-8r2t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:15:04.116873 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.116843    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-util" pod "1489bd54-2e77-417f-a157-f83840cd1e74" (UID: "1489bd54-2e77-417f-a157-f83840cd1e74"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:15:04.118289 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.118266    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-util" pod "f6e8d4ca-17c7-4c23-bc88-934fa6824168" (UID: "f6e8d4ca-17c7-4c23-bc88-934fa6824168"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:15:04.212784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.212749    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqjv9\" (UniqueName: \"kubernetes.io/projected/1489bd54-2e77-417f-a157-f83840cd1e74-kube-api-access-rqjv9\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:04.212784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.212784    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2t4\" (UniqueName: \"kubernetes.io/projected/f6e8d4ca-17c7-4c23-bc88-934fa6824168-kube-api-access-8r2t4\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:04.212979 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.212800    5112 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-bundle\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:04.212979 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.212814    5112 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e8d4ca-17c7-4c23-bc88-934fa6824168-util\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:04.212979 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.212827    5112 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-bundle\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:04.212979 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.212840    5112 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1489bd54-2e77-417f-a157-f83840cd1e74-util\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:04.819262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.819156    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:15:04.819945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.819922    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst"
Nov 03 16:15:04.820053 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.819921    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst" event={"ID":"f6e8d4ca-17c7-4c23-bc88-934fa6824168","Type":"ContainerDied","Data":"7931b682a7d71e2731e5604f189f1c6e3faee4c874f2b3b493f5bd0b0eaca9ef"}
Nov 03 16:15:04.820105 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.820052    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7931b682a7d71e2731e5604f189f1c6e3faee4c874f2b3b493f5bd0b0eaca9ef"
Nov 03 16:15:04.822186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.821918    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt"
Nov 03 16:15:04.822186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.821930    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71q4wqt" event={"ID":"1489bd54-2e77-417f-a157-f83840cd1e74","Type":"ContainerDied","Data":"853bffd37fb9f5ca8e5097554b4474a8198f1ac605d5747a8746adb1a596e887"}
Nov 03 16:15:04.822186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.821972    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="853bffd37fb9f5ca8e5097554b4474a8198f1ac605d5747a8746adb1a596e887"
Nov 03 16:15:04.945843 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:04.945819    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:15:05.120381 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.120296    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5342e8ae-34c1-4f6a-b671-24546d54380e-kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/5342e8ae-34c1-4f6a-b671-24546d54380e-kube-api-access-8mb4d\") pod \"5342e8ae-34c1-4f6a-b671-24546d54380e\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") "
Nov 03 16:15:05.120381 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.120368    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-bundle\") pod \"5342e8ae-34c1-4f6a-b671-24546d54380e\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") "
Nov 03 16:15:05.120607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.120391    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-util\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-util\") pod \"5342e8ae-34c1-4f6a-b671-24546d54380e\" (UID: \"5342e8ae-34c1-4f6a-b671-24546d54380e\") "
Nov 03 16:15:05.120861 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.120837    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-bundle" pod "5342e8ae-34c1-4f6a-b671-24546d54380e" (UID: "5342e8ae-34c1-4f6a-b671-24546d54380e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:15:05.122538 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.122515    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5342e8ae-34c1-4f6a-b671-24546d54380e-kube-api-access-8mb4d" pod "5342e8ae-34c1-4f6a-b671-24546d54380e" (UID: "5342e8ae-34c1-4f6a-b671-24546d54380e"). InnerVolumeSpecName "kube-api-access-8mb4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:15:05.126060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.126032    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-util" pod "5342e8ae-34c1-4f6a-b671-24546d54380e" (UID: "5342e8ae-34c1-4f6a-b671-24546d54380e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:15:05.220935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.220905    5112 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-bundle\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:05.220935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.220928    5112 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5342e8ae-34c1-4f6a-b671-24546d54380e-util\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:05.220935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.220938    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/5342e8ae-34c1-4f6a-b671-24546d54380e-kube-api-access-8mb4d\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:05.829211 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.829166    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84" event={"ID":"5342e8ae-34c1-4f6a-b671-24546d54380e","Type":"ContainerDied","Data":"aea3257ba37455736740a642b305233756d84fdeb2f30a77d4b272aac6e8c747"}
Nov 03 16:15:05.829565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.829230    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aea3257ba37455736740a642b305233756d84fdeb2f30a77d4b272aac6e8c747"
Nov 03 16:15:05.829565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:05.829176    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg5g84"
Nov 03 16:15:07.882229 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:15:07.882171    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:15:08.897946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.897911    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"]
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898239    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerName="extract"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898251    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerName="extract"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898267    5112 cpu_manager.go:414] "containerMap: removing container" podUID="1489bd54-2e77-417f-a157-f83840cd1e74" containerName="extract"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898272    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1489bd54-2e77-417f-a157-f83840cd1e74" containerName="extract"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898280    5112 cpu_manager.go:414] "containerMap: removing container" podUID="1489bd54-2e77-417f-a157-f83840cd1e74" containerName="pull"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898286    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1489bd54-2e77-417f-a157-f83840cd1e74" containerName="pull"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898301    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerName="util"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898306    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerName="util"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898311    5112 cpu_manager.go:414] "containerMap: removing container" podUID="1489bd54-2e77-417f-a157-f83840cd1e74" containerName="util"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898316    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1489bd54-2e77-417f-a157-f83840cd1e74" containerName="util"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898324    5112 cpu_manager.go:414] "containerMap: removing container" podUID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerName="extract"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898328    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerName="extract"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898340    5112 cpu_manager.go:414] "containerMap: removing container" podUID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerName="pull"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898344    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerName="pull"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898349    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerName="pull"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898354    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerName="pull"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898362    5112 cpu_manager.go:414] "containerMap: removing container" podUID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerName="util"
Nov 03 16:15:08.898363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898367    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerName="util"
Nov 03 16:15:08.898876 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898415    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e8d4ca-17c7-4c23-bc88-934fa6824168" containerName="extract"
Nov 03 16:15:08.898876 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898424    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="1489bd54-2e77-417f-a157-f83840cd1e74" containerName="extract"
Nov 03 16:15:08.898876 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.898431    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="5342e8ae-34c1-4f6a-b671-24546d54380e" containerName="extract"
Nov 03 16:15:08.942772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.942745    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:08.999096 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.999065    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 16:15:08.999310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.999112    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 16:15:08.999310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:08.999177    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 16:15:09.044793 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.044760    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 16:15:09.044962 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.044794    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rbhfv\""
Nov 03 16:15:09.044962 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.044794    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 16:15:09.051423 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.051391    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"]
Nov 03 16:15:09.054402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.054375    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b9052a-0244-4350-abb7-aaed2c7cb4e3-tmp\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.054513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.054410    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.054563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.054504    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-457kk\" (UniqueName: \"kubernetes.io/projected/24b9052a-0244-4350-abb7-aaed2c7cb4e3-kube-api-access-457kk\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.054618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.054601    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24b9052a-0244-4350-abb7-aaed2c7cb4e3-manifests\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.054683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.054634    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.054683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.054661    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.156136 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.156048    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-457kk\" (UniqueName: \"kubernetes.io/projected/24b9052a-0244-4350-abb7-aaed2c7cb4e3-kube-api-access-457kk\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.156136 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.156108    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24b9052a-0244-4350-abb7-aaed2c7cb4e3-manifests\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.156136 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.156133    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.156467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.156157    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.156467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.156194    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b9052a-0244-4350-abb7-aaed2c7cb4e3-tmp\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.156467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.156277    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.156669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.156644    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24b9052a-0244-4350-abb7-aaed2c7cb4e3-manifests\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.158903 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.158876    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.159004 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.158913    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.159004 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.158921    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b9052a-0244-4350-abb7-aaed2c7cb4e3-tmp\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.159090 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.159032    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24b9052a-0244-4350-abb7-aaed2c7cb4e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.213158 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.213122    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-457kk\" (UniqueName: \"kubernetes.io/projected/24b9052a-0244-4350-abb7-aaed2c7cb4e3-kube-api-access-457kk\") pod \"opendatahub-operator-controller-manager-7d94989496-mjpkm\" (UID: \"24b9052a-0244-4350-abb7-aaed2c7cb4e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.291807 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.291770    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:09.878754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:09.878724    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"]
Nov 03 16:15:09.879640 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:09.879616    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b9052a_0244_4350_abb7_aaed2c7cb4e3.slice/crio-04a1bfa9eaea1a35876009ceaf7b1c166c11d6d72babdb25891792447019a24c WatchSource:0}: Error finding container 04a1bfa9eaea1a35876009ceaf7b1c166c11d6d72babdb25891792447019a24c: Status 404 returned error can't find the container with id 04a1bfa9eaea1a35876009ceaf7b1c166c11d6d72babdb25891792447019a24c
Nov 03 16:15:10.851539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:10.851502    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm" event={"ID":"24b9052a-0244-4350-abb7-aaed2c7cb4e3","Type":"ContainerStarted","Data":"04a1bfa9eaea1a35876009ceaf7b1c166c11d6d72babdb25891792447019a24c"}
Nov 03 16:15:15.702934 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.702903    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"]
Nov 03 16:15:15.706437 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.706419    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:15.709992 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.709969    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpldb\" (UniqueName: \"kubernetes.io/projected/1cf57e6a-d4e4-4484-94fd-acb883261ec4-kube-api-access-kpldb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2\" (UID: \"1cf57e6a-d4e4-4484-94fd-acb883261ec4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:15.710082 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.710015    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cf57e6a-d4e4-4484-94fd-acb883261ec4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2\" (UID: \"1cf57e6a-d4e4-4484-94fd-acb883261ec4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:15.748639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.748617    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"]
Nov 03 16:15:15.794466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.794442    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 16:15:15.794600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.794502    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 16:15:15.794600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.794519    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qvnwg\""
Nov 03 16:15:15.810913 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.810881    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpldb\" (UniqueName: \"kubernetes.io/projected/1cf57e6a-d4e4-4484-94fd-acb883261ec4-kube-api-access-kpldb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2\" (UID: \"1cf57e6a-d4e4-4484-94fd-acb883261ec4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:15.811015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.810958    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cf57e6a-d4e4-4484-94fd-acb883261ec4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2\" (UID: \"1cf57e6a-d4e4-4484-94fd-acb883261ec4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:15.811340 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.811308    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cf57e6a-d4e4-4484-94fd-acb883261ec4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2\" (UID: \"1cf57e6a-d4e4-4484-94fd-acb883261ec4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:15.875016 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.874989    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpldb\" (UniqueName: \"kubernetes.io/projected/1cf57e6a-d4e4-4484-94fd-acb883261ec4-kube-api-access-kpldb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2\" (UID: \"1cf57e6a-d4e4-4484-94fd-acb883261ec4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:15.876500 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:15.876469    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm" event={"ID":"24b9052a-0244-4350-abb7-aaed2c7cb4e3","Type":"ContainerStarted","Data":"be4dcf308ef2f1c60f0d87076fa836953e4b1f82f083079152d648faf4d868dd"}
Nov 03 16:15:16.018766 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:16.018688    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"
Nov 03 16:15:16.477101 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:16.477060    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2"]
Nov 03 16:15:16.480289 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:16.480260    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf57e6a_d4e4_4484_94fd_acb883261ec4.slice/crio-bf83301065f3902bef79d186355c11b3b10498f9d3440de0cf93ab9c71839d9a WatchSource:0}: Error finding container bf83301065f3902bef79d186355c11b3b10498f9d3440de0cf93ab9c71839d9a: Status 404 returned error can't find the container with id bf83301065f3902bef79d186355c11b3b10498f9d3440de0cf93ab9c71839d9a
Nov 03 16:15:16.881679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:16.881643    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2" event={"ID":"1cf57e6a-d4e4-4484-94fd-acb883261ec4","Type":"ContainerStarted","Data":"bf83301065f3902bef79d186355c11b3b10498f9d3440de0cf93ab9c71839d9a"}
Nov 03 16:15:16.883163 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:16.883134    5112 generic.go:356] "Generic (PLEG): container finished" podID="24b9052a-0244-4350-abb7-aaed2c7cb4e3" containerID="be4dcf308ef2f1c60f0d87076fa836953e4b1f82f083079152d648faf4d868dd" exitCode=0
Nov 03 16:15:16.883300 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:16.883209    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm" event={"ID":"24b9052a-0244-4350-abb7-aaed2c7cb4e3","Type":"ContainerDied","Data":"be4dcf308ef2f1c60f0d87076fa836953e4b1f82f083079152d648faf4d868dd"}
Nov 03 16:15:17.889085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:17.889000    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm" event={"ID":"24b9052a-0244-4350-abb7-aaed2c7cb4e3","Type":"ContainerStarted","Data":"91d7822aac30ed953d193231a50f74d1d4048db189fb5f3ca9249b57e9b5cc15"}
Nov 03 16:15:17.889575 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:17.889112    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:18.039011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:18.038895    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm" podStartSLOduration=2.913298528 podStartE2EDuration="10.038881855s" totalImagesPullingTime="6.679421733s" totalInitContainerRuntime="446.161594ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:18.036657468 +0000 UTC m=+279.615961342" watchObservedRunningTime="2026-11-03 16:15:18.038881855 +0000 UTC m=+279.618185710"
Nov 03 16:15:21.909905 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:21.909867    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2" event={"ID":"1cf57e6a-d4e4-4484-94fd-acb883261ec4","Type":"ContainerStarted","Data":"b70206212173757036203f72a5b90dcb33d694f9a137064a5a68240bea675f64"}
Nov 03 16:15:22.124879 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:22.124819    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xvmd2" podStartSLOduration=2.425036216 podStartE2EDuration="7.12480531s" totalImagesPullingTime="4.699769094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:22.077194912 +0000 UTC m=+283.656498766" watchObservedRunningTime="2026-11-03 16:15:22.12480531 +0000 UTC m=+283.704109167"
Nov 03 16:15:28.897474 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:28.897437    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d94989496-mjpkm"
Nov 03 16:15:32.384563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.384527    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"]
Nov 03 16:15:32.389879 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.389862    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:32.423803 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.423769    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9j5\" (UniqueName: \"kubernetes.io/projected/a419eb0c-a1ab-4deb-b341-951569bfbdad-kube-api-access-2m9j5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-t6wbm\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:32.423936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.423847    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a419eb0c-a1ab-4deb-b341-951569bfbdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-t6wbm\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:32.437325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.437301    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"]
Nov 03 16:15:32.482141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.482116    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 16:15:32.482141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.482119    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bmzc9\""
Nov 03 16:15:32.482331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.482154    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 16:15:32.525104 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.525063    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9j5\" (UniqueName: \"kubernetes.io/projected/a419eb0c-a1ab-4deb-b341-951569bfbdad-kube-api-access-2m9j5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-t6wbm\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:32.525322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.525141    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a419eb0c-a1ab-4deb-b341-951569bfbdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-t6wbm\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:32.525551 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.525532    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a419eb0c-a1ab-4deb-b341-951569bfbdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-t6wbm\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:32.583170 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.583136    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9j5\" (UniqueName: \"kubernetes.io/projected/a419eb0c-a1ab-4deb-b341-951569bfbdad-kube-api-access-2m9j5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-t6wbm\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:32.702356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:32.702326    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:33.169942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:33.169913    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"]
Nov 03 16:15:33.172222 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:33.172164    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda419eb0c_a1ab_4deb_b341_951569bfbdad.slice/crio-19c2d5e62d9683206d8be384a648dc4e744aace05712b515b75ea39a704c4962 WatchSource:0}: Error finding container 19c2d5e62d9683206d8be384a648dc4e744aace05712b515b75ea39a704c4962: Status 404 returned error can't find the container with id 19c2d5e62d9683206d8be384a648dc4e744aace05712b515b75ea39a704c4962
Nov 03 16:15:33.971442 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:33.971403    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm" event={"ID":"a419eb0c-a1ab-4deb-b341-951569bfbdad","Type":"ContainerStarted","Data":"19c2d5e62d9683206d8be384a648dc4e744aace05712b515b75ea39a704c4962"}
Nov 03 16:15:37.337550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.337511    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"]
Nov 03 16:15:37.341040 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.341012    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"
Nov 03 16:15:37.438473 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.438442    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jk75t\""
Nov 03 16:15:37.440314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.440285    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"]
Nov 03 16:15:37.459029 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.459000    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqrk\" (UniqueName: \"kubernetes.io/projected/d08ca72c-70da-42cd-8ea9-b44a5f078c2d-kube-api-access-rwqrk\") pod \"authorino-operator-7b7bb4bc77-tggc5\" (UID: \"d08ca72c-70da-42cd-8ea9-b44a5f078c2d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"
Nov 03 16:15:37.559885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.559848    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqrk\" (UniqueName: \"kubernetes.io/projected/d08ca72c-70da-42cd-8ea9-b44a5f078c2d-kube-api-access-rwqrk\") pod \"authorino-operator-7b7bb4bc77-tggc5\" (UID: \"d08ca72c-70da-42cd-8ea9-b44a5f078c2d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"
Nov 03 16:15:37.617714 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.617640    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqrk\" (UniqueName: \"kubernetes.io/projected/d08ca72c-70da-42cd-8ea9-b44a5f078c2d-kube-api-access-rwqrk\") pod \"authorino-operator-7b7bb4bc77-tggc5\" (UID: \"d08ca72c-70da-42cd-8ea9-b44a5f078c2d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"
Nov 03 16:15:37.657135 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:37.657103    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"
Nov 03 16:15:37.892846 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:15:37.892763    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:15:38.047611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:38.047578    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"]
Nov 03 16:15:38.079103 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:38.079068    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08ca72c_70da_42cd_8ea9_b44a5f078c2d.slice/crio-ce1b4cad4223cb014cc57ed9e0dee08dc930ec7bb49b7e2f74ba523e86131ffd WatchSource:0}: Error finding container ce1b4cad4223cb014cc57ed9e0dee08dc930ec7bb49b7e2f74ba523e86131ffd: Status 404 returned error can't find the container with id ce1b4cad4223cb014cc57ed9e0dee08dc930ec7bb49b7e2f74ba523e86131ffd
Nov 03 16:15:38.561007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:38.560957    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:15:38.562033 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:38.562008    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:15:38.574791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:38.574769    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:15:38.575554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:38.575534    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:15:38.996741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:38.996704    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5" event={"ID":"d08ca72c-70da-42cd-8ea9-b44a5f078c2d","Type":"ContainerStarted","Data":"ce1b4cad4223cb014cc57ed9e0dee08dc930ec7bb49b7e2f74ba523e86131ffd"}
Nov 03 16:15:41.434367 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.434338    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"]
Nov 03 16:15:41.437999 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.437982    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"
Nov 03 16:15:41.490534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.490510    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ft66\" (UniqueName: \"kubernetes.io/projected/f78e2948-1cd5-4098-84d4-ae66c985fdba-kube-api-access-8ft66\") pod \"limitador-operator-controller-manager-5778bb4449-5d2fl\" (UID: \"f78e2948-1cd5-4098-84d4-ae66c985fdba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"
Nov 03 16:15:41.496240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.496191    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"]
Nov 03 16:15:41.537638 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.537580    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5kvc2\""
Nov 03 16:15:41.591877 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.591844    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ft66\" (UniqueName: \"kubernetes.io/projected/f78e2948-1cd5-4098-84d4-ae66c985fdba-kube-api-access-8ft66\") pod \"limitador-operator-controller-manager-5778bb4449-5d2fl\" (UID: \"f78e2948-1cd5-4098-84d4-ae66c985fdba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"
Nov 03 16:15:41.651052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.651023    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ft66\" (UniqueName: \"kubernetes.io/projected/f78e2948-1cd5-4098-84d4-ae66c985fdba-kube-api-access-8ft66\") pod \"limitador-operator-controller-manager-5778bb4449-5d2fl\" (UID: \"f78e2948-1cd5-4098-84d4-ae66c985fdba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"
Nov 03 16:15:41.751101 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:41.751049    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"
Nov 03 16:15:42.018118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:42.018083    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5" event={"ID":"d08ca72c-70da-42cd-8ea9-b44a5f078c2d","Type":"ContainerStarted","Data":"66194616608c594264bbf458e0bbba90cfc8c4d8c279c3de8b0d419985d0e3da"}
Nov 03 16:15:42.018318 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:42.018250    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"
Nov 03 16:15:42.128327 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:42.128302    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"]
Nov 03 16:15:42.131732 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:42.131689    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78e2948_1cd5_4098_84d4_ae66c985fdba.slice/crio-23888eab54a0226eb4e3909ee3df80bdaa51fa601d676d2347c7645039e45389 WatchSource:0}: Error finding container 23888eab54a0226eb4e3909ee3df80bdaa51fa601d676d2347c7645039e45389: Status 404 returned error can't find the container with id 23888eab54a0226eb4e3909ee3df80bdaa51fa601d676d2347c7645039e45389
Nov 03 16:15:42.134034 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:42.134013    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:15:42.172946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:42.172895    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5" podStartSLOduration=1.99511395 podStartE2EDuration="5.172875874s" totalImagesPullingTime="3.177761924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:42.128936822 +0000 UTC m=+303.708240698" watchObservedRunningTime="2026-11-03 16:15:42.172875874 +0000 UTC m=+303.752179730"
Nov 03 16:15:43.023983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:43.023954    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm" event={"ID":"a419eb0c-a1ab-4deb-b341-951569bfbdad","Type":"ContainerStarted","Data":"8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35"}
Nov 03 16:15:43.024399 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:43.024059    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:43.025408 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:43.025384    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl" event={"ID":"f78e2948-1cd5-4098-84d4-ae66c985fdba","Type":"ContainerStarted","Data":"23888eab54a0226eb4e3909ee3df80bdaa51fa601d676d2347c7645039e45389"}
Nov 03 16:15:43.238909 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:43.238809    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm" podStartSLOduration=1.4963971169999999 podStartE2EDuration="11.238792978s" totalImagesPullingTime="9.742395861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:43.190473529 +0000 UTC m=+304.769777402" watchObservedRunningTime="2026-11-03 16:15:43.238792978 +0000 UTC m=+304.818096910"
Nov 03 16:15:45.037480 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:45.037373    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl" event={"ID":"f78e2948-1cd5-4098-84d4-ae66c985fdba","Type":"ContainerStarted","Data":"92ec417b7882a93ed137c9b67db18abc07252c96587b1f49266051fb1f89608b"}
Nov 03 16:15:45.037880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:45.037569    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"
Nov 03 16:15:45.157841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:45.157776    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl" podStartSLOduration=1.561155568 podStartE2EDuration="4.157760632s" totalImagesPullingTime="2.596605064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:45.15705489 +0000 UTC m=+306.736358742" watchObservedRunningTime="2026-11-03 16:15:45.157760632 +0000 UTC m=+306.737064486"
Nov 03 16:15:48.375917 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.375879    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"]
Nov 03 16:15:48.379873 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.379853    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.431622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.431592    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-t5nzs\""
Nov 03 16:15:48.458548 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458513    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458548 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458552    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458581    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trzp\" (UniqueName: \"kubernetes.io/projected/7feef304-fd59-4aa7-adb6-adbeacd388b6-kube-api-access-9trzp\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458703    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458780    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7feef304-fd59-4aa7-adb6-adbeacd388b6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458806    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458904 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458878    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458944 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458906    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.458983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.458940    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.479596 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.479559    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"]
Nov 03 16:15:48.559789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.559731    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.559789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.559808    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7feef304-fd59-4aa7-adb6-adbeacd388b6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.559837    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.559875    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.559901    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.559936    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.560006    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.560030    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.560066    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trzp\" (UniqueName: \"kubernetes.io/projected/7feef304-fd59-4aa7-adb6-adbeacd388b6-kube-api-access-9trzp\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560456 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.560303    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.560623    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7feef304-fd59-4aa7-adb6-adbeacd388b6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.560826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.560804    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.561135 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.561089    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.561293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.561184    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.563004 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.562979    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.563139 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.563116    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.618280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.618238    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7feef304-fd59-4aa7-adb6-adbeacd388b6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.618428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.618252    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trzp\" (UniqueName: \"kubernetes.io/projected/7feef304-fd59-4aa7-adb6-adbeacd388b6-kube-api-access-9trzp\") pod \"data-science-gateway-data-science-gateway-class-6455b4886f59wpd\" (UID: \"7feef304-fd59-4aa7-adb6-adbeacd388b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:48.694344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:48.694316    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:49.120382 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.120290    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"]
Nov 03 16:15:49.123498 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:49.123467    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7feef304_fd59_4aa7_adb6_adbeacd388b6.slice/crio-b447288b2e6801f251056d1daacb027aacb79ad92ae71a44498b0e819ee5d56d WatchSource:0}: Error finding container b447288b2e6801f251056d1daacb027aacb79ad92ae71a44498b0e819ee5d56d: Status 404 returned error can't find the container with id b447288b2e6801f251056d1daacb027aacb79ad92ae71a44498b0e819ee5d56d
Nov 03 16:15:49.125971 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.125936    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:49.126059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.126011    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:49.126059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.126042    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:15:49.690990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.690962    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rpnq5"]
Nov 03 16:15:49.694605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.694586    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rpnq5"
Nov 03 16:15:49.750128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.750088    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rpnq5"]
Nov 03 16:15:49.750346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.750318    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zd89p\""
Nov 03 16:15:49.770650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.770623    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sgb\" (UniqueName: \"kubernetes.io/projected/ded36530-5a79-4970-a5b7-b95b974d317f-kube-api-access-45sgb\") pod \"authorino-57f7f8659f-rpnq5\" (UID: \"ded36530-5a79-4970-a5b7-b95b974d317f\") " pod="kuadrant-system/authorino-57f7f8659f-rpnq5"
Nov 03 16:15:49.871320 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.871283    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sgb\" (UniqueName: \"kubernetes.io/projected/ded36530-5a79-4970-a5b7-b95b974d317f-kube-api-access-45sgb\") pod \"authorino-57f7f8659f-rpnq5\" (UID: \"ded36530-5a79-4970-a5b7-b95b974d317f\") " pod="kuadrant-system/authorino-57f7f8659f-rpnq5"
Nov 03 16:15:49.935449 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:49.935410    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sgb\" (UniqueName: \"kubernetes.io/projected/ded36530-5a79-4970-a5b7-b95b974d317f-kube-api-access-45sgb\") pod \"authorino-57f7f8659f-rpnq5\" (UID: \"ded36530-5a79-4970-a5b7-b95b974d317f\") " pod="kuadrant-system/authorino-57f7f8659f-rpnq5"
Nov 03 16:15:50.016766 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:50.016692    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rpnq5"
Nov 03 16:15:50.062992 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:50.062953    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd" event={"ID":"7feef304-fd59-4aa7-adb6-adbeacd388b6","Type":"ContainerStarted","Data":"ae7cc9fc1e74ee8d68c44ec69846252e0541596df732c3a529ea00cdc43ac100"}
Nov 03 16:15:50.063178 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:50.063049    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd" event={"ID":"7feef304-fd59-4aa7-adb6-adbeacd388b6","Type":"ContainerStarted","Data":"b447288b2e6801f251056d1daacb027aacb79ad92ae71a44498b0e819ee5d56d"}
Nov 03 16:15:50.183765 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:50.183704    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd" podStartSLOduration=2.183689407 podStartE2EDuration="2.183689407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:50.181866716 +0000 UTC m=+311.761170574" watchObservedRunningTime="2026-11-03 16:15:50.183689407 +0000 UTC m=+311.762993332"
Nov 03 16:15:50.382564 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:50.382539    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rpnq5"]
Nov 03 16:15:50.384955 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:50.384927    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podded36530_5a79_4970_a5b7_b95b974d317f.slice/crio-b80d70ebdd399e7523ca7f75bd84e9c5dc2bd7ed3849645d4be18e8a31662b5f WatchSource:0}: Error finding container b80d70ebdd399e7523ca7f75bd84e9c5dc2bd7ed3849645d4be18e8a31662b5f: Status 404 returned error can't find the container with id b80d70ebdd399e7523ca7f75bd84e9c5dc2bd7ed3849645d4be18e8a31662b5f
Nov 03 16:15:50.694814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:50.694778    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:50.699956 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:50.699931    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:51.068948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:51.068844    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rpnq5" event={"ID":"ded36530-5a79-4970-a5b7-b95b974d317f","Type":"ContainerStarted","Data":"b80d70ebdd399e7523ca7f75bd84e9c5dc2bd7ed3849645d4be18e8a31662b5f"}
Nov 03 16:15:51.069260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:51.069234    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:51.070427 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:51.070399    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886f59wpd"
Nov 03 16:15:52.550971 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.550912    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"]
Nov 03 16:15:52.554667 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.554644    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.598483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.598442    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.598689 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.598501    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlsw\" (UniqueName: \"kubernetes.io/projected/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-kube-api-access-wqlsw\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.598689 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.598618    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.612041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.611984    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"]
Nov 03 16:15:52.655238 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.655176    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 16:15:52.655410 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.655267    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jjqwv\""
Nov 03 16:15:52.656023 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.656006    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 16:15:52.699974 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.699942    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.699974 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.699975    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlsw\" (UniqueName: \"kubernetes.io/projected/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-kube-api-access-wqlsw\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.700175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.699998    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.700640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.700622    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.702730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.702707    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.757450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.757409    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlsw\" (UniqueName: \"kubernetes.io/projected/3fbdbf34-dc5c-4f9c-87a1-c6b391294237-kube-api-access-wqlsw\") pod \"kuadrant-console-plugin-66c4fb575d-6j829\" (UID: \"3fbdbf34-dc5c-4f9c-87a1-c6b391294237\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:52.867770 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:52.867683    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"
Nov 03 16:15:53.028655 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:53.028620    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tggc5"
Nov 03 16:15:53.252641 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:53.252610    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829"]
Nov 03 16:15:53.254779 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:53.254747    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fbdbf34_dc5c_4f9c_87a1_c6b391294237.slice/crio-8c8ff34cde2af00735bfe20c8d85c1e0cad50296a5e8baf61f3a0a20c9dc2c17 WatchSource:0}: Error finding container 8c8ff34cde2af00735bfe20c8d85c1e0cad50296a5e8baf61f3a0a20c9dc2c17: Status 404 returned error can't find the container with id 8c8ff34cde2af00735bfe20c8d85c1e0cad50296a5e8baf61f3a0a20c9dc2c17
Nov 03 16:15:54.033785 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.033721    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:15:54.093880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.093843    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rpnq5" event={"ID":"ded36530-5a79-4970-a5b7-b95b974d317f","Type":"ContainerStarted","Data":"18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea"}
Nov 03 16:15:54.095236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.095188    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829" event={"ID":"3fbdbf34-dc5c-4f9c-87a1-c6b391294237","Type":"ContainerStarted","Data":"8c8ff34cde2af00735bfe20c8d85c1e0cad50296a5e8baf61f3a0a20c9dc2c17"}
Nov 03 16:15:54.271019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.270964    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-rpnq5" podStartSLOduration=2.492285105 podStartE2EDuration="5.270950133s" totalImagesPullingTime="2.778665028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:54.270081224 +0000 UTC m=+315.849385102" watchObservedRunningTime="2026-11-03 16:15:54.270950133 +0000 UTC m=+315.850253987"
Nov 03 16:15:54.520953 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.520920    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"]
Nov 03 16:15:54.525784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.525756    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:54.585941 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.585905    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"]
Nov 03 16:15:54.617394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.617356    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wdmmk\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:54.617688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.617597    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4vw\" (UniqueName: \"kubernetes.io/projected/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-kube-api-access-fp4vw\") pod \"kuadrant-operator-controller-manager-678d87766f-wdmmk\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:54.718841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.718799    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4vw\" (UniqueName: \"kubernetes.io/projected/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-kube-api-access-fp4vw\") pod \"kuadrant-operator-controller-manager-678d87766f-wdmmk\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:54.719015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.718853    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wdmmk\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:54.719229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.719193    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wdmmk\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:54.775018 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.774948    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4vw\" (UniqueName: \"kubernetes.io/projected/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-kube-api-access-fp4vw\") pod \"kuadrant-operator-controller-manager-678d87766f-wdmmk\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:54.842493 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:54.842458    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:55.308679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:55.308645    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"]
Nov 03 16:15:55.310389 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:15:55.310352    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bb58f6_9b4f_4db3_9c87_3b1deb5de984.slice/crio-3fe56ecf921661f28c8ecdc2a20a322adb8f93b22caae3b4113bd08a46f55930 WatchSource:0}: Error finding container 3fe56ecf921661f28c8ecdc2a20a322adb8f93b22caae3b4113bd08a46f55930: Status 404 returned error can't find the container with id 3fe56ecf921661f28c8ecdc2a20a322adb8f93b22caae3b4113bd08a46f55930
Nov 03 16:15:56.045211 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:56.045162    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5d2fl"
Nov 03 16:15:56.111246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:56.110721    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk" event={"ID":"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984","Type":"ContainerStarted","Data":"fdebe5e9de11e9e206adf5ddf6d277c5c20dcef81608c248b0362d8989b4fe7f"}
Nov 03 16:15:56.111246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:56.110786    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk" event={"ID":"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984","Type":"ContainerStarted","Data":"3fe56ecf921661f28c8ecdc2a20a322adb8f93b22caae3b4113bd08a46f55930"}
Nov 03 16:15:56.111246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:56.110913    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:15:56.289516 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:56.289458    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk" podStartSLOduration=2.289442115 podStartE2EDuration="2.289442115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:56.287928736 +0000 UTC m=+317.867232614" watchObservedRunningTime="2026-11-03 16:15:56.289442115 +0000 UTC m=+317.868745969"
Nov 03 16:15:57.218964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:57.218930    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rpnq5"]
Nov 03 16:15:57.219449 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:57.219099    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-rpnq5" podUID="ded36530-5a79-4970-a5b7-b95b974d317f" containerName="authorino" containerID="cri-o://18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea" gracePeriod=30
Nov 03 16:15:57.502250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:57.502191    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rpnq5"
Nov 03 16:15:57.648922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:57.648890    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded36530-5a79-4970-a5b7-b95b974d317f-kube-api-access-45sgb\" (UniqueName: \"kubernetes.io/projected/ded36530-5a79-4970-a5b7-b95b974d317f-kube-api-access-45sgb\") pod \"ded36530-5a79-4970-a5b7-b95b974d317f\" (UID: \"ded36530-5a79-4970-a5b7-b95b974d317f\") "
Nov 03 16:15:57.651063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:57.651034    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded36530-5a79-4970-a5b7-b95b974d317f-kube-api-access-45sgb" pod "ded36530-5a79-4970-a5b7-b95b974d317f" (UID: "ded36530-5a79-4970-a5b7-b95b974d317f"). InnerVolumeSpecName "kube-api-access-45sgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:15:57.750117 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:57.750034    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45sgb\" (UniqueName: \"kubernetes.io/projected/ded36530-5a79-4970-a5b7-b95b974d317f-kube-api-access-45sgb\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:15:58.122491 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.122393    5112 generic.go:356] "Generic (PLEG): container finished" podID="ded36530-5a79-4970-a5b7-b95b974d317f" containerID="18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea" exitCode=0
Nov 03 16:15:58.122491 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.122443    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rpnq5"
Nov 03 16:15:58.122722 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.122484    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rpnq5" event={"ID":"ded36530-5a79-4970-a5b7-b95b974d317f","Type":"ContainerDied","Data":"18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea"}
Nov 03 16:15:58.122722 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.122525    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rpnq5" event={"ID":"ded36530-5a79-4970-a5b7-b95b974d317f","Type":"ContainerDied","Data":"b80d70ebdd399e7523ca7f75bd84e9c5dc2bd7ed3849645d4be18e8a31662b5f"}
Nov 03 16:15:58.122722 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.122544    5112 scope.go:122] "RemoveContainer" containerID="18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea"
Nov 03 16:15:58.133069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.133050    5112 scope.go:122] "RemoveContainer" containerID="18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea"
Nov 03 16:15:58.133381 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:15:58.133361    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea\": container with ID starting with 18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea not found: ID does not exist" containerID="18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea"
Nov 03 16:15:58.133458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.133395    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea"} err="failed to get container status \"18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea\": rpc error: code = NotFound desc = could not find container \"18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea\": container with ID starting with 18cb5d8a61d7a0c51d2875c3976d68c3e00e9ecadc2351c8add8e8f724270fea not found: ID does not exist"
Nov 03 16:15:58.281488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.281451    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rpnq5"]
Nov 03 16:15:58.293230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.293180    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rpnq5"]
Nov 03 16:15:58.550639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:15:58.550594    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded36530-5a79-4970-a5b7-b95b974d317f" path="/var/lib/kubelet/pods/ded36530-5a79-4970-a5b7-b95b974d317f/volumes"
Nov 03 16:16:01.150458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:01.150419    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829" event={"ID":"3fbdbf34-dc5c-4f9c-87a1-c6b391294237","Type":"ContainerStarted","Data":"ddb12e8c921df4cc0b9ff382329a119bf030ef26b0ae5d863a091d23a2822c1f"}
Nov 03 16:16:01.264168 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:01.264111    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6j829" podStartSLOduration=1.735156377 podStartE2EDuration="9.264097548s" totalImagesPullingTime="7.528941171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:01.262979528 +0000 UTC m=+322.842283402" watchObservedRunningTime="2026-11-03 16:16:01.264097548 +0000 UTC m=+322.843401402"
Nov 03 16:16:07.117831 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.117796    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 16:16:07.329521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.329475    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"]
Nov 03 16:16:07.329738 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.329714    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm" podUID="a419eb0c-a1ab-4deb-b341-951569bfbdad" containerName="manager" containerID="cri-o://8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35" gracePeriod=10
Nov 03 16:16:07.597299 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.597273    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:16:07.741737 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.741706    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a419eb0c-a1ab-4deb-b341-951569bfbdad-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a419eb0c-a1ab-4deb-b341-951569bfbdad-extensions-socket-volume\") pod \"a419eb0c-a1ab-4deb-b341-951569bfbdad\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") "
Nov 03 16:16:07.741943 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.741792    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a419eb0c-a1ab-4deb-b341-951569bfbdad-kube-api-access-2m9j5\" (UniqueName: \"kubernetes.io/projected/a419eb0c-a1ab-4deb-b341-951569bfbdad-kube-api-access-2m9j5\") pod \"a419eb0c-a1ab-4deb-b341-951569bfbdad\" (UID: \"a419eb0c-a1ab-4deb-b341-951569bfbdad\") "
Nov 03 16:16:07.742412 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.742372    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a419eb0c-a1ab-4deb-b341-951569bfbdad-extensions-socket-volume" pod "a419eb0c-a1ab-4deb-b341-951569bfbdad" (UID: "a419eb0c-a1ab-4deb-b341-951569bfbdad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:16:07.743996 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.743971    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a419eb0c-a1ab-4deb-b341-951569bfbdad-kube-api-access-2m9j5" pod "a419eb0c-a1ab-4deb-b341-951569bfbdad" (UID: "a419eb0c-a1ab-4deb-b341-951569bfbdad"). InnerVolumeSpecName "kube-api-access-2m9j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:16:07.842913 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.842881    5112 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a419eb0c-a1ab-4deb-b341-951569bfbdad-extensions-socket-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:16:07.842913 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:07.842909    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9j5\" (UniqueName: \"kubernetes.io/projected/a419eb0c-a1ab-4deb-b341-951569bfbdad-kube-api-access-2m9j5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:16:07.883762 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:16:07.883731    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:16:08.183945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.183912    5112 generic.go:356] "Generic (PLEG): container finished" podID="a419eb0c-a1ab-4deb-b341-951569bfbdad" containerID="8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35" exitCode=0
Nov 03 16:16:08.184467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.183955    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm" event={"ID":"a419eb0c-a1ab-4deb-b341-951569bfbdad","Type":"ContainerDied","Data":"8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35"}
Nov 03 16:16:08.184467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.183978    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"
Nov 03 16:16:08.184467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.183992    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm" event={"ID":"a419eb0c-a1ab-4deb-b341-951569bfbdad","Type":"ContainerDied","Data":"19c2d5e62d9683206d8be384a648dc4e744aace05712b515b75ea39a704c4962"}
Nov 03 16:16:08.184467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.184013    5112 scope.go:122] "RemoveContainer" containerID="8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35"
Nov 03 16:16:08.193753 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.193736    5112 scope.go:122] "RemoveContainer" containerID="8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35"
Nov 03 16:16:08.194028 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:16:08.194007    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35\": container with ID starting with 8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35 not found: ID does not exist" containerID="8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35"
Nov 03 16:16:08.194109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.194037    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35"} err="failed to get container status \"8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35\": rpc error: code = NotFound desc = could not find container \"8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35\": container with ID starting with 8b715e8e6f168c125e11f72a92882f4dee505d19f557bb1675128cc377923c35 not found: ID does not exist"
Nov 03 16:16:08.359567 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.359527    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"]
Nov 03 16:16:08.406591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.406549    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-t6wbm"]
Nov 03 16:16:08.547226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:08.547135    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a419eb0c-a1ab-4deb-b341-951569bfbdad" path="/var/lib/kubelet/pods/a419eb0c-a1ab-4deb-b341-951569bfbdad/volumes"
Nov 03 16:16:14.352888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.352854    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"]
Nov 03 16:16:14.353322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.353257    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a419eb0c-a1ab-4deb-b341-951569bfbdad" containerName="manager"
Nov 03 16:16:14.353322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.353272    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a419eb0c-a1ab-4deb-b341-951569bfbdad" containerName="manager"
Nov 03 16:16:14.353322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.353292    5112 cpu_manager.go:414] "containerMap: removing container" podUID="ded36530-5a79-4970-a5b7-b95b974d317f" containerName="authorino"
Nov 03 16:16:14.353322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.353298    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded36530-5a79-4970-a5b7-b95b974d317f" containerName="authorino"
Nov 03 16:16:14.353443 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.353378    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a419eb0c-a1ab-4deb-b341-951569bfbdad" containerName="manager"
Nov 03 16:16:14.353443 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.353389    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded36530-5a79-4970-a5b7-b95b974d317f" containerName="authorino"
Nov 03 16:16:14.404627 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.404602    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.449538 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.449506    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"]
Nov 03 16:16:14.494344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.494306    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-8plcq\""
Nov 03 16:16:14.494344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.494329    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 16:16:14.494344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.494341    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 16:16:14.500802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.500778    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7a492f5-71b4-4348-886d-72bafe60d156-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.500914 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.500808    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7a492f5-71b4-4348-886d-72bafe60d156-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.500914 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.500826    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7a492f5-71b4-4348-886d-72bafe60d156-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.501030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.500965    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxz5\" (UniqueName: \"kubernetes.io/projected/f7a492f5-71b4-4348-886d-72bafe60d156-kube-api-access-vnxz5\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.589398 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.589368    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 16:16:14.602032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.602005    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7a492f5-71b4-4348-886d-72bafe60d156-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.602142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.602042    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7a492f5-71b4-4348-886d-72bafe60d156-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.602142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.602061    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7a492f5-71b4-4348-886d-72bafe60d156-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.602142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.602100    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxz5\" (UniqueName: \"kubernetes.io/projected/f7a492f5-71b4-4348-886d-72bafe60d156-kube-api-access-vnxz5\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.602487 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.602467    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7a492f5-71b4-4348-886d-72bafe60d156-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.602522 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.602493    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7a492f5-71b4-4348-886d-72bafe60d156-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.602935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.602896    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7a492f5-71b4-4348-886d-72bafe60d156-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.659684 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.659649    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxz5\" (UniqueName: \"kubernetes.io/projected/f7a492f5-71b4-4348-886d-72bafe60d156-kube-api-access-vnxz5\") pod \"kserve-module-controller-manager-7566cf5fc8-5tm5k\" (UID: \"f7a492f5-71b4-4348-886d-72bafe60d156\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:14.738992 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:14.738963    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:15.172568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:15.172546    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"]
Nov 03 16:16:15.174544 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:16:15.174497    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a492f5_71b4_4348_886d_72bafe60d156.slice/crio-3f42711f87eb78105d36b99d39f0c7341032bfbb4a65ba7b96e050e2ae9f383c WatchSource:0}: Error finding container 3f42711f87eb78105d36b99d39f0c7341032bfbb4a65ba7b96e050e2ae9f383c: Status 404 returned error can't find the container with id 3f42711f87eb78105d36b99d39f0c7341032bfbb4a65ba7b96e050e2ae9f383c
Nov 03 16:16:15.226259 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:15.226223    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k" event={"ID":"f7a492f5-71b4-4348-886d-72bafe60d156","Type":"ContainerStarted","Data":"3f42711f87eb78105d36b99d39f0c7341032bfbb4a65ba7b96e050e2ae9f383c"}
Nov 03 16:16:20.252881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:20.252854    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k" event={"ID":"f7a492f5-71b4-4348-886d-72bafe60d156","Type":"ContainerStarted","Data":"3057764b0b1e94decbc1355d12c0b77e130cff96e36724784738d55e0de05407"}
Nov 03 16:16:20.253229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:20.252917    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:20.404546 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:20.404490    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k" podStartSLOduration=1.395812175 podStartE2EDuration="6.404476848s" totalImagesPullingTime="5.008664673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:20.403654853 +0000 UTC m=+341.982958708" watchObservedRunningTime="2026-11-03 16:16:20.404476848 +0000 UTC m=+341.983780694"
Nov 03 16:16:34.525618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.525580    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"]
Nov 03 16:16:34.547309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.547270    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:34.572731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.572697    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1937961f-b133-4de5-b7cd-7011593da92a-data\") pod \"postgres-7cdbf6dc44-sldd2\" (UID: \"1937961f-b133-4de5-b7cd-7011593da92a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:34.572898 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.572778    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hjf\" (UniqueName: \"kubernetes.io/projected/1937961f-b133-4de5-b7cd-7011593da92a-kube-api-access-g9hjf\") pod \"postgres-7cdbf6dc44-sldd2\" (UID: \"1937961f-b133-4de5-b7cd-7011593da92a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:34.584320 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.584172    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"]
Nov 03 16:16:34.597854 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.597822    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 16:16:34.629660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.629614    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nvmjd\""
Nov 03 16:16:34.629660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.629657    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 16:16:34.629871 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.629655    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 16:16:34.673590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.673544    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1937961f-b133-4de5-b7cd-7011593da92a-data\") pod \"postgres-7cdbf6dc44-sldd2\" (UID: \"1937961f-b133-4de5-b7cd-7011593da92a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:34.673956 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.673925    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hjf\" (UniqueName: \"kubernetes.io/projected/1937961f-b133-4de5-b7cd-7011593da92a-kube-api-access-g9hjf\") pod \"postgres-7cdbf6dc44-sldd2\" (UID: \"1937961f-b133-4de5-b7cd-7011593da92a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:34.674123 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.674103    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1937961f-b133-4de5-b7cd-7011593da92a-data\") pod \"postgres-7cdbf6dc44-sldd2\" (UID: \"1937961f-b133-4de5-b7cd-7011593da92a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:34.730691 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.730657    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hjf\" (UniqueName: \"kubernetes.io/projected/1937961f-b133-4de5-b7cd-7011593da92a-kube-api-access-g9hjf\") pod \"postgres-7cdbf6dc44-sldd2\" (UID: \"1937961f-b133-4de5-b7cd-7011593da92a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:34.865428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:34.865327    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:35.244582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:35.244549    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"]
Nov 03 16:16:35.245970 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:16:35.245945    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1937961f_b133_4de5_b7cd_7011593da92a.slice/crio-c78997ec03d5fd5a1c80be8870e07ab8ea094f96a927b3d31972aa726aba744c WatchSource:0}: Error finding container c78997ec03d5fd5a1c80be8870e07ab8ea094f96a927b3d31972aa726aba744c: Status 404 returned error can't find the container with id c78997ec03d5fd5a1c80be8870e07ab8ea094f96a927b3d31972aa726aba744c
Nov 03 16:16:35.334219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:35.334158    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2" event={"ID":"1937961f-b133-4de5-b7cd-7011593da92a","Type":"ContainerStarted","Data":"c78997ec03d5fd5a1c80be8870e07ab8ea094f96a927b3d31972aa726aba744c"}
Nov 03 16:16:37.884113 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:16:37.884076    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:16:41.260574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:41.260533    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5tm5k"
Nov 03 16:16:44.101367 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:44.101332    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 16:16:44.401107 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:44.401066    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2" event={"ID":"1937961f-b133-4de5-b7cd-7011593da92a","Type":"ContainerStarted","Data":"73380b2d2620a2e2996490496515b631d0f1e941cbf2899bbc723a3e64d95d64"}
Nov 03 16:16:44.401388 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:44.401225    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:16:44.513939 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:44.513881    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2" podStartSLOduration=1.709478116 podStartE2EDuration="10.51386727s" totalImagesPullingTime="8.804389154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:44.512806495 +0000 UTC m=+366.092110361" watchObservedRunningTime="2026-11-03 16:16:44.51386727 +0000 UTC m=+366.093171125"
Nov 03 16:16:50.436342 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:16:50.436263    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sldd2"
Nov 03 16:17:07.883639 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:17:07.883598    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:17:22.670744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.670709    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hvcc9"]
Nov 03 16:17:22.676070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.676050    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:22.734037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.734012    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hvcc9"]
Nov 03 16:17:22.776444 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.776418    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xdjqv\""
Nov 03 16:17:22.776444 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.776418    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 16:17:22.797483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.797452    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b00b85c8-024d-4cba-be13-1963655418ed-cert\") pod \"kserve-controller-manager-776cc66d47-hvcc9\" (UID: \"b00b85c8-024d-4cba-be13-1963655418ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:22.797641 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.797498    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvdr\" (UniqueName: \"kubernetes.io/projected/b00b85c8-024d-4cba-be13-1963655418ed-kube-api-access-bpvdr\") pod \"kserve-controller-manager-776cc66d47-hvcc9\" (UID: \"b00b85c8-024d-4cba-be13-1963655418ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:22.898527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.898495    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b00b85c8-024d-4cba-be13-1963655418ed-cert\") pod \"kserve-controller-manager-776cc66d47-hvcc9\" (UID: \"b00b85c8-024d-4cba-be13-1963655418ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:22.898707 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.898545    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvdr\" (UniqueName: \"kubernetes.io/projected/b00b85c8-024d-4cba-be13-1963655418ed-kube-api-access-bpvdr\") pod \"kserve-controller-manager-776cc66d47-hvcc9\" (UID: \"b00b85c8-024d-4cba-be13-1963655418ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:22.900850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.900830    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b00b85c8-024d-4cba-be13-1963655418ed-cert\") pod \"kserve-controller-manager-776cc66d47-hvcc9\" (UID: \"b00b85c8-024d-4cba-be13-1963655418ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:22.954870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.954809    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvdr\" (UniqueName: \"kubernetes.io/projected/b00b85c8-024d-4cba-be13-1963655418ed-kube-api-access-bpvdr\") pod \"kserve-controller-manager-776cc66d47-hvcc9\" (UID: \"b00b85c8-024d-4cba-be13-1963655418ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:22.957912 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.957889    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2r7rn"]
Nov 03 16:17:22.961797 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.961779    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:22.988344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:22.988324    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:23.016681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.016654    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2r7rn"]
Nov 03 16:17:23.035582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.035532    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zfnnw\""
Nov 03 16:17:23.035582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.035532    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 16:17:23.035719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.035594    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 16:17:23.100449 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.100421    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-cert\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.100604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.100488    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-metrics-certs\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.100604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.100587    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8xp\" (UniqueName: \"kubernetes.io/projected/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-kube-api-access-cj8xp\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.201445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.201412    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8xp\" (UniqueName: \"kubernetes.io/projected/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-kube-api-access-cj8xp\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.201591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.201478    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-cert\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.201591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.201565    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-metrics-certs\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.203941 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.203916    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-metrics-certs\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.204041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.203922    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-cert\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.258017 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.257969    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8xp\" (UniqueName: \"kubernetes.io/projected/d3a3c2ca-0044-4f06-9178-8baa1a312c2c-kube-api-access-cj8xp\") pod \"odh-model-controller-769875db56-2r7rn\" (UID: \"d3a3c2ca-0044-4f06-9178-8baa1a312c2c\") " pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.275845 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.275824    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:23.360568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.360544    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hvcc9"]
Nov 03 16:17:23.362286 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:17:23.362258    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00b85c8_024d_4cba_be13_1963655418ed.slice/crio-642848c6943d25c85e699975ecd5ffc1c0a630413fac19ec0dd19908b4fffe6e WatchSource:0}: Error finding container 642848c6943d25c85e699975ecd5ffc1c0a630413fac19ec0dd19908b4fffe6e: Status 404 returned error can't find the container with id 642848c6943d25c85e699975ecd5ffc1c0a630413fac19ec0dd19908b4fffe6e
Nov 03 16:17:23.596169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.596102    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9" event={"ID":"b00b85c8-024d-4cba-be13-1963655418ed","Type":"ContainerStarted","Data":"642848c6943d25c85e699975ecd5ffc1c0a630413fac19ec0dd19908b4fffe6e"}
Nov 03 16:17:23.647383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:23.647357    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2r7rn"]
Nov 03 16:17:23.649061 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:17:23.649027    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a3c2ca_0044_4f06_9178_8baa1a312c2c.slice/crio-7c4099ca634b2e0fcbc9ce225c2bf105f6ad08df2481b94df64ba11661fe8e50 WatchSource:0}: Error finding container 7c4099ca634b2e0fcbc9ce225c2bf105f6ad08df2481b94df64ba11661fe8e50: Status 404 returned error can't find the container with id 7c4099ca634b2e0fcbc9ce225c2bf105f6ad08df2481b94df64ba11661fe8e50
Nov 03 16:17:24.602315 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:24.602273    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2r7rn" event={"ID":"d3a3c2ca-0044-4f06-9178-8baa1a312c2c","Type":"ContainerStarted","Data":"7c4099ca634b2e0fcbc9ce225c2bf105f6ad08df2481b94df64ba11661fe8e50"}
Nov 03 16:17:27.619600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:27.619564    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2r7rn" event={"ID":"d3a3c2ca-0044-4f06-9178-8baa1a312c2c","Type":"ContainerStarted","Data":"332a1e972d18079d6cc2899ba816ecf7b98f5941a2ef48d9aea830e563163e70"}
Nov 03 16:17:27.620034 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:27.619748    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:27.621150 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:27.621125    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9" event={"ID":"b00b85c8-024d-4cba-be13-1963655418ed","Type":"ContainerStarted","Data":"91cb91f8f32b6639920cd4a0ab2e857ef3a85c78ac334013412d8d9a7904aa9c"}
Nov 03 16:17:27.621277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:27.621259    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:27.732844 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:27.732791    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2r7rn" podStartSLOduration=2.504538834 podStartE2EDuration="5.732779228s" totalImagesPullingTime="3.228240394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:27.732128891 +0000 UTC m=+409.311432764" watchObservedRunningTime="2026-11-03 16:17:27.732779228 +0000 UTC m=+409.312083083"
Nov 03 16:17:27.844169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:27.844116    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9" podStartSLOduration=2.39648791 podStartE2EDuration="5.844098781s" totalImagesPullingTime="3.447610871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:27.842559241 +0000 UTC m=+409.421863103" watchObservedRunningTime="2026-11-03 16:17:27.844098781 +0000 UTC m=+409.423402637"
Nov 03 16:17:37.884773 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:17:37.884733    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:17:38.627472 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:38.627444    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2r7rn"
Nov 03 16:17:40.780633 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.780595    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"]
Nov 03 16:17:40.785323 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.785302    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.840600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.840575    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"]
Nov 03 16:17:40.855357 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.855333    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8k8\" (UniqueName: \"kubernetes.io/projected/2a233e70-eb1b-4140-be64-5572ee9afd45-kube-api-access-bp8k8\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.855464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.855380    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-metrics-certs\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.855464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.855416    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-maas-api-tls\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.855544 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.855508    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a233e70-eb1b-4140-be64-5572ee9afd45-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.863926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.863901    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 16:17:40.864011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.863939    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7k42d\""
Nov 03 16:17:40.884963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.884944    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 16:17:40.956526 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.956505    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-maas-api-tls\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.956616 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.956533    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a233e70-eb1b-4140-be64-5572ee9afd45-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.956616 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.956591    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8k8\" (UniqueName: \"kubernetes.io/projected/2a233e70-eb1b-4140-be64-5572ee9afd45-kube-api-access-bp8k8\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.956725 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.956619    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-metrics-certs\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.957391 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.957365    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a233e70-eb1b-4140-be64-5572ee9afd45-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.959041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.959024    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-metrics-certs\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:40.959115 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:40.959061    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-maas-api-tls\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:41.012776 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:41.012751    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8k8\" (UniqueName: \"kubernetes.io/projected/2a233e70-eb1b-4140-be64-5572ee9afd45-kube-api-access-bp8k8\") pod \"maas-api-6d669c7fcd-kjx2m\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:41.098702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:41.098647    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:41.474406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:41.474384    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"]
Nov 03 16:17:41.475989 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:17:41.475959    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a233e70_eb1b_4140_be64_5572ee9afd45.slice/crio-1b4f2a3cb4fe7534e8be43f52d2841c913909a91b9a4f566206119ece518cb8b WatchSource:0}: Error finding container 1b4f2a3cb4fe7534e8be43f52d2841c913909a91b9a4f566206119ece518cb8b: Status 404 returned error can't find the container with id 1b4f2a3cb4fe7534e8be43f52d2841c913909a91b9a4f566206119ece518cb8b
Nov 03 16:17:41.689073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:41.689037    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m" event={"ID":"2a233e70-eb1b-4140-be64-5572ee9afd45","Type":"ContainerStarted","Data":"1b4f2a3cb4fe7534e8be43f52d2841c913909a91b9a4f566206119ece518cb8b"}
Nov 03 16:17:45.710356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:45.710315    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m" event={"ID":"2a233e70-eb1b-4140-be64-5572ee9afd45","Type":"ContainerStarted","Data":"beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1"}
Nov 03 16:17:45.710715 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:45.710404    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:45.823822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:45.823778    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m" podStartSLOduration=2.552225888 podStartE2EDuration="5.823765713s" totalImagesPullingTime="3.271539825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:45.82268895 +0000 UTC m=+427.401992805" watchObservedRunningTime="2026-11-03 16:17:45.823765713 +0000 UTC m=+427.403069567"
Nov 03 16:17:51.721130 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:51.721100    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:17:53.004679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.004642    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cb88f96bb-jfl5l"]
Nov 03 16:17:53.009572 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.009548    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l"
Nov 03 16:17:53.063246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.063195    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cb88f96bb-jfl5l"]
Nov 03 16:17:53.092404 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.092368    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t547d\" (UniqueName: \"kubernetes.io/projected/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9-kube-api-access-t547d\") pod \"authorino-7cb88f96bb-jfl5l\" (UID: \"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9\") " pod="kuadrant-system/authorino-7cb88f96bb-jfl5l"
Nov 03 16:17:53.108414 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.108374    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zd89p\""
Nov 03 16:17:53.193293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.193238    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t547d\" (UniqueName: \"kubernetes.io/projected/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9-kube-api-access-t547d\") pod \"authorino-7cb88f96bb-jfl5l\" (UID: \"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9\") " pod="kuadrant-system/authorino-7cb88f96bb-jfl5l"
Nov 03 16:17:53.250235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.250180    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t547d\" (UniqueName: \"kubernetes.io/projected/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9-kube-api-access-t547d\") pod \"authorino-7cb88f96bb-jfl5l\" (UID: \"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9\") " pod="kuadrant-system/authorino-7cb88f96bb-jfl5l"
Nov 03 16:17:53.321785 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.321710    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l"
Nov 03 16:17:53.438592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.438560    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cb88f96bb-jfl5l"]
Nov 03 16:17:53.685945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.685918    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cb88f96bb-jfl5l"]
Nov 03 16:17:53.689392 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:17:53.689359    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda03e2f8e_7ff4_4c63_b95f_3f034535d9d9.slice/crio-f583e994f510e40ed1f58a553a46448c25209ecba49fb4e7b400185a5cc1887b WatchSource:0}: Error finding container f583e994f510e40ed1f58a553a46448c25209ecba49fb4e7b400185a5cc1887b: Status 404 returned error can't find the container with id f583e994f510e40ed1f58a553a46448c25209ecba49fb4e7b400185a5cc1887b
Nov 03 16:17:53.750904 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:53.750872    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l" event={"ID":"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9","Type":"ContainerStarted","Data":"f583e994f510e40ed1f58a553a46448c25209ecba49fb4e7b400185a5cc1887b"}
Nov 03 16:17:55.762424 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.762329    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l" event={"ID":"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9","Type":"ContainerStarted","Data":"d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb"}
Nov 03 16:17:55.762424 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.762368    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l" podUID="a03e2f8e-7ff4-4c63-b95f-3f034535d9d9" containerName="authorino" containerID="cri-o://d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb" gracePeriod=30
Nov 03 16:17:55.763977 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.763952    5112 generic.go:356] "Generic (PLEG): container finished" podID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerID="50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51" exitCode=0
Nov 03 16:17:55.764097 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.763995    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" event={"ID":"725cb813-4d0c-4922-b79c-ec7c2624d5a7","Type":"ContainerDied","Data":"50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51"}
Nov 03 16:17:55.764306 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.764293    5112 scope.go:122] "RemoveContainer" containerID="50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51"
Nov 03 16:17:55.765571 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.765535    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:17:55.765662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.765610    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:17:55.765662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.765647    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:17:55.874288 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:55.874246    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l" podStartSLOduration=2.111210742 podStartE2EDuration="3.874231213s" totalImagesPullingTime="1.763020471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:17:55.872488136 +0000 UTC m=+437.451792011" watchObservedRunningTime="2026-11-03 16:17:55.874231213 +0000 UTC m=+437.453535061"
Nov 03 16:17:56.037294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.037263    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l"
Nov 03 16:17:56.121895 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.121855    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9-kube-api-access-t547d\" (UniqueName: \"kubernetes.io/projected/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9-kube-api-access-t547d\") pod \"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9\" (UID: \"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9\") "
Nov 03 16:17:56.123947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.123913    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9-kube-api-access-t547d" pod "a03e2f8e-7ff4-4c63-b95f-3f034535d9d9" (UID: "a03e2f8e-7ff4-4c63-b95f-3f034535d9d9"). InnerVolumeSpecName "kube-api-access-t547d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:17:56.223046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.223016    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t547d\" (UniqueName: \"kubernetes.io/projected/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9-kube-api-access-t547d\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:17:56.784942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.784908    5112 generic.go:356] "Generic (PLEG): container finished" podID="a03e2f8e-7ff4-4c63-b95f-3f034535d9d9" containerID="d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb" exitCode=0
Nov 03 16:17:56.785466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.784941    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l" event={"ID":"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9","Type":"ContainerDied","Data":"d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb"}
Nov 03 16:17:56.785466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.784984    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l" event={"ID":"a03e2f8e-7ff4-4c63-b95f-3f034535d9d9","Type":"ContainerDied","Data":"f583e994f510e40ed1f58a553a46448c25209ecba49fb4e7b400185a5cc1887b"}
Nov 03 16:17:56.785466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.785001    5112 scope.go:122] "RemoveContainer" containerID="d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb"
Nov 03 16:17:56.785466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.784949    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb88f96bb-jfl5l"
Nov 03 16:17:56.787510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.787489    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" event={"ID":"725cb813-4d0c-4922-b79c-ec7c2624d5a7","Type":"ContainerStarted","Data":"6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b"}
Nov 03 16:17:56.795027 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.795010    5112 scope.go:122] "RemoveContainer" containerID="d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb"
Nov 03 16:17:56.795368 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:17:56.795341    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb\": container with ID starting with d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb not found: ID does not exist" containerID="d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb"
Nov 03 16:17:56.795453 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:56.795380    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb"} err="failed to get container status \"d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb\": rpc error: code = NotFound desc = could not find container \"d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb\": container with ID starting with d01fda8dd55efb527be694c45b2218c1c3ce16540a593950cb97a0d9305cadeb not found: ID does not exist"
Nov 03 16:17:57.076221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:57.071678    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cb88f96bb-jfl5l"]
Nov 03 16:17:57.080872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:57.080832    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cb88f96bb-jfl5l"]
Nov 03 16:17:57.379527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:57.379432    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:17:57.379865 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:57.379824    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sbbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 16:17:57.379986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:57.379879    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 16:17:58.379700 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:58.379659    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sbbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 16:17:58.380302 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:58.380266    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 16:17:58.548463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:58.548432    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03e2f8e-7ff4-4c63-b95f-3f034535d9d9" path="/var/lib/kubelet/pods/a03e2f8e-7ff4-4c63-b95f-3f034535d9d9/volumes"
Nov 03 16:17:58.630729 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:58.630655    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hvcc9"
Nov 03 16:17:59.378524 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:59.378488    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sbbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 16:17:59.378766 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:17:59.378739    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 16:18:00.380602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:00.380566    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sbbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 16:18:00.381575 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:00.381167    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 16:18:01.381621 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:01.381575    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sbbf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 03 16:18:01.382123 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:01.381639    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 03 16:18:02.380835 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:02.380804    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:18:02.381112 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:02.381090    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:18:02.381681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:02.381666    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:18:02.892707 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:02.892670    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"]
Nov 03 16:18:02.892933 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:02.892899    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m" podUID="2a233e70-eb1b-4140-be64-5572ee9afd45" containerName="maas-api" containerID="cri-o://beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1" gracePeriod=30
Nov 03 16:18:03.141287 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.141255    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:18:03.309729 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.309694    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-metrics-certs\") pod \"2a233e70-eb1b-4140-be64-5572ee9afd45\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") "
Nov 03 16:18:03.309927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.309777    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a233e70-eb1b-4140-be64-5572ee9afd45-kube-api-access-bp8k8\" (UniqueName: \"kubernetes.io/projected/2a233e70-eb1b-4140-be64-5572ee9afd45-kube-api-access-bp8k8\") pod \"2a233e70-eb1b-4140-be64-5572ee9afd45\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") "
Nov 03 16:18:03.309927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.309821    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a233e70-eb1b-4140-be64-5572ee9afd45-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a233e70-eb1b-4140-be64-5572ee9afd45-openshift-service-ca\") pod \"2a233e70-eb1b-4140-be64-5572ee9afd45\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") "
Nov 03 16:18:03.309927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.309878    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-maas-api-tls\") pod \"2a233e70-eb1b-4140-be64-5572ee9afd45\" (UID: \"2a233e70-eb1b-4140-be64-5572ee9afd45\") "
Nov 03 16:18:03.310243 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.310216    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a233e70-eb1b-4140-be64-5572ee9afd45-openshift-service-ca" pod "2a233e70-eb1b-4140-be64-5572ee9afd45" (UID: "2a233e70-eb1b-4140-be64-5572ee9afd45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:18:03.312088 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.312053    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-maas-api-tls" pod "2a233e70-eb1b-4140-be64-5572ee9afd45" (UID: "2a233e70-eb1b-4140-be64-5572ee9afd45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:03.312249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.312111    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-metrics-certs" pod "2a233e70-eb1b-4140-be64-5572ee9afd45" (UID: "2a233e70-eb1b-4140-be64-5572ee9afd45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:18:03.312249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.312185    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a233e70-eb1b-4140-be64-5572ee9afd45-kube-api-access-bp8k8" pod "2a233e70-eb1b-4140-be64-5572ee9afd45" (UID: "2a233e70-eb1b-4140-be64-5572ee9afd45"). InnerVolumeSpecName "kube-api-access-bp8k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:18:03.410971 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.410941    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:03.410971 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.410967    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8k8\" (UniqueName: \"kubernetes.io/projected/2a233e70-eb1b-4140-be64-5572ee9afd45-kube-api-access-bp8k8\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:03.410971 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.410979    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a233e70-eb1b-4140-be64-5572ee9afd45-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:03.411459 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.410987    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a233e70-eb1b-4140-be64-5572ee9afd45-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:18:03.825707 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.825673    5112 generic.go:356] "Generic (PLEG): container finished" podID="2a233e70-eb1b-4140-be64-5572ee9afd45" containerID="beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1" exitCode=0
Nov 03 16:18:03.825882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.825720    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"
Nov 03 16:18:03.825882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.825766    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m" event={"ID":"2a233e70-eb1b-4140-be64-5572ee9afd45","Type":"ContainerDied","Data":"beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1"}
Nov 03 16:18:03.825882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.825825    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m" event={"ID":"2a233e70-eb1b-4140-be64-5572ee9afd45","Type":"ContainerDied","Data":"1b4f2a3cb4fe7534e8be43f52d2841c913909a91b9a4f566206119ece518cb8b"}
Nov 03 16:18:03.825882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.825851    5112 scope.go:122] "RemoveContainer" containerID="beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1"
Nov 03 16:18:03.835147 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.835129    5112 scope.go:122] "RemoveContainer" containerID="beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1"
Nov 03 16:18:03.835439 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:18:03.835417    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1\": container with ID starting with beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1 not found: ID does not exist" containerID="beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1"
Nov 03 16:18:03.835483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.835449    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1"} err="failed to get container status \"beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1\": rpc error: code = NotFound desc = could not find container \"beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1\": container with ID starting with beb4479559e7545707e7fca0cae99a35edaefeb690d8ec90aa800df83d1a13c1 not found: ID does not exist"
Nov 03 16:18:03.985740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:03.985702    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"]
Nov 03 16:18:04.032887 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:04.032856    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kjx2m"]
Nov 03 16:18:04.548099 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:18:04.548067    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a233e70-eb1b-4140-be64-5572ee9afd45" path="/var/lib/kubelet/pods/2a233e70-eb1b-4140-be64-5572ee9afd45/volumes"
Nov 03 16:18:07.881758 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:18:07.881726    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:18:37.882536 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:18:37.882486    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:19:07.881533 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:19:07.881501    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:19:32.773988 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.773957    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"]
Nov 03 16:19:32.774475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.774378    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2a233e70-eb1b-4140-be64-5572ee9afd45" containerName="maas-api"
Nov 03 16:19:32.774475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.774391    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a233e70-eb1b-4140-be64-5572ee9afd45" containerName="maas-api"
Nov 03 16:19:32.774475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.774420    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a03e2f8e-7ff4-4c63-b95f-3f034535d9d9" containerName="authorino"
Nov 03 16:19:32.774475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.774425    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03e2f8e-7ff4-4c63-b95f-3f034535d9d9" containerName="authorino"
Nov 03 16:19:32.774602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.774483    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a233e70-eb1b-4140-be64-5572ee9afd45" containerName="maas-api"
Nov 03 16:19:32.774602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.774497    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03e2f8e-7ff4-4c63-b95f-3f034535d9d9" containerName="authorino"
Nov 03 16:19:32.777712 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.777689    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.844498 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.844466    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwwz\" (UniqueName: \"kubernetes.io/projected/465a697d-a817-4a5d-af72-11a56526571f-kube-api-access-qzwwz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.844670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.844512    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.844670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.844530    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.844759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.844678    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465a697d-a817-4a5d-af72-11a56526571f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.844795 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.844755    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.844833 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.844807    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.881149 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.881110    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 16:19:32.881339 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.881166    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fh4dr\""
Nov 03 16:19:32.881339 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.881214    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 16:19:32.881339 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.881189    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 16:19:32.883374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.883357    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"]
Nov 03 16:19:32.945669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.945631    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.945860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.945690    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.945860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.945721    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwwz\" (UniqueName: \"kubernetes.io/projected/465a697d-a817-4a5d-af72-11a56526571f-kube-api-access-qzwwz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.945860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.945763    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.945860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.945785    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.945860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.945835    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465a697d-a817-4a5d-af72-11a56526571f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.946116 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.946046    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.946176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.946131    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.946176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.946161    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.948084 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.948063    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465a697d-a817-4a5d-af72-11a56526571f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:32.948382 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:32.948364    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465a697d-a817-4a5d-af72-11a56526571f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:33.007566 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:33.007537    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwwz\" (UniqueName: \"kubernetes.io/projected/465a697d-a817-4a5d-af72-11a56526571f-kube-api-access-qzwwz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z\" (UID: \"465a697d-a817-4a5d-af72-11a56526571f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:33.090797 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:33.090713    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:33.467954 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:33.467926    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"]
Nov 03 16:19:33.469140 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:19:33.469109    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465a697d_a817_4a5d_af72_11a56526571f.slice/crio-b614d13a0b3c4e448f1e7bb3fef68354fa67502a755d87baaea0676f349f0d6e WatchSource:0}: Error finding container b614d13a0b3c4e448f1e7bb3fef68354fa67502a755d87baaea0676f349f0d6e: Status 404 returned error can't find the container with id b614d13a0b3c4e448f1e7bb3fef68354fa67502a755d87baaea0676f349f0d6e
Nov 03 16:19:34.267242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:34.267192    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z" event={"ID":"465a697d-a817-4a5d-af72-11a56526571f","Type":"ContainerStarted","Data":"b614d13a0b3c4e448f1e7bb3fef68354fa67502a755d87baaea0676f349f0d6e"}
Nov 03 16:19:36.279123 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:36.279084    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z" event={"ID":"465a697d-a817-4a5d-af72-11a56526571f","Type":"ContainerStarted","Data":"c0552e42cf87592602fc389ca2c35d8181e52925c889bd1f7db8e52bfbb34178"}
Nov 03 16:19:36.409780 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:36.409717    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z" podStartSLOduration=2.051911645 podStartE2EDuration="4.409698101s" totalImagesPullingTime="2.357786456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:36.407784086 +0000 UTC m=+537.987087971" watchObservedRunningTime="2026-11-03 16:19:36.409698101 +0000 UTC m=+537.989001958"
Nov 03 16:19:37.091094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:37.091062    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:37.883245 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:19:37.883181    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:19:42.116049 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.116017    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"]
Nov 03 16:19:42.121542 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.121512    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.217553 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.217528    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 16:19:42.222964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.222940    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"]
Nov 03 16:19:42.225191 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.225164    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.225300 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.225222    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.225340 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.225315    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.225422 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.225340    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298vg\" (UniqueName: \"kubernetes.io/projected/32d8a9c3-50b8-4487-951a-0963519e47b5-kube-api-access-298vg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.225477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.225459    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.225518 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.225488    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8a9c3-50b8-4487-951a-0963519e47b5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326089 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326058    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8a9c3-50b8-4487-951a-0963519e47b5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326097    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326118    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326154    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326171    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298vg\" (UniqueName: \"kubernetes.io/projected/32d8a9c3-50b8-4487-951a-0963519e47b5-kube-api-access-298vg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326253    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326522    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326589    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.326660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.326613    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.328390 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.328368    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32d8a9c3-50b8-4487-951a-0963519e47b5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.328655 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.328637    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32d8a9c3-50b8-4487-951a-0963519e47b5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.390995 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.390940    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-298vg\" (UniqueName: \"kubernetes.io/projected/32d8a9c3-50b8-4487-951a-0963519e47b5-kube-api-access-298vg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4v77c\" (UID: \"32d8a9c3-50b8-4487-951a-0963519e47b5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.437374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.437350    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:42.816929 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:42.816894    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"]
Nov 03 16:19:42.818274 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:19:42.818242    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d8a9c3_50b8_4487_951a_0963519e47b5.slice/crio-14ac1cf4b3f2f6aa241a0464a92d3bd4cc39a1b8ae2a341b72439b338d3854dc WatchSource:0}: Error finding container 14ac1cf4b3f2f6aa241a0464a92d3bd4cc39a1b8ae2a341b72439b338d3854dc: Status 404 returned error can't find the container with id 14ac1cf4b3f2f6aa241a0464a92d3bd4cc39a1b8ae2a341b72439b338d3854dc
Nov 03 16:19:43.091265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:43.091176    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:43.103430 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:43.103406    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:43.318343 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:43.318312    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c" event={"ID":"32d8a9c3-50b8-4487-951a-0963519e47b5","Type":"ContainerStarted","Data":"14ac1cf4b3f2f6aa241a0464a92d3bd4cc39a1b8ae2a341b72439b338d3854dc"}
Nov 03 16:19:43.329398 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:43.329374    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z"
Nov 03 16:19:44.324969 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:44.324923    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c" event={"ID":"32d8a9c3-50b8-4487-951a-0963519e47b5","Type":"ContainerStarted","Data":"d1d0aa455b79b1b0743602fdd777241bd10c41381c38825b236e475216a6ca98"}
Nov 03 16:19:44.437497 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:44.437470    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:44.440503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:44.440461    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c" podStartSLOduration=1.809111643 podStartE2EDuration="2.440449277s" totalImagesPullingTime="631.337634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:44.438705601 +0000 UTC m=+546.018009472" watchObservedRunningTime="2026-11-03 16:19:44.440449277 +0000 UTC m=+546.019753133"
Nov 03 16:19:52.440082 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:52.439432    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:52.452920 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:52.452896    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:53.382611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:53.382577    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c"
Nov 03 16:19:53.981559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:53.981527    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"]
Nov 03 16:19:53.985153 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:53.985132    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.036891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.036867    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 16:19:54.084088 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.084062    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"]
Nov 03 16:19:54.132420 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.132395    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.132573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.132427    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.132573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.132445    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfqp\" (UniqueName: \"kubernetes.io/projected/26d95fe7-b2ba-4872-9587-647d5488e5a2-kube-api-access-hgfqp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.132660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.132583    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.132754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.132737    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.132828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.132769    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d95fe7-b2ba-4872-9587-647d5488e5a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233508 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233441    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233508 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233474    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233508 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233492    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfqp\" (UniqueName: \"kubernetes.io/projected/26d95fe7-b2ba-4872-9587-647d5488e5a2-kube-api-access-hgfqp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233529    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233670    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233837 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233725    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d95fe7-b2ba-4872-9587-647d5488e5a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233893 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233860    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.233945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233910    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.234005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.233983    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.235930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.235903    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d95fe7-b2ba-4872-9587-647d5488e5a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.236299 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.236279    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d95fe7-b2ba-4872-9587-647d5488e5a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.289174 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.289156    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfqp\" (UniqueName: \"kubernetes.io/projected/26d95fe7-b2ba-4872-9587-647d5488e5a2-kube-api-access-hgfqp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn\" (UID: \"26d95fe7-b2ba-4872-9587-647d5488e5a2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.298035 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.298016    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:19:54.667878 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:54.667854    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"]
Nov 03 16:19:54.670036 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:19:54.670012    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d95fe7_b2ba_4872_9587_647d5488e5a2.slice/crio-b85da5f71391bb8c8ae86be5aeb1ff4ec5309b4740fdf61c16085dd5286911c8 WatchSource:0}: Error finding container b85da5f71391bb8c8ae86be5aeb1ff4ec5309b4740fdf61c16085dd5286911c8: Status 404 returned error can't find the container with id b85da5f71391bb8c8ae86be5aeb1ff4ec5309b4740fdf61c16085dd5286911c8
Nov 03 16:19:55.383807 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:55.383770    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn" event={"ID":"26d95fe7-b2ba-4872-9587-647d5488e5a2","Type":"ContainerStarted","Data":"2f096bee1970c339cd1a11ec31478b4f4afdcda091f5aed46cf607e311038f4e"}
Nov 03 16:19:55.384179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:55.383831    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn" event={"ID":"26d95fe7-b2ba-4872-9587-647d5488e5a2","Type":"ContainerStarted","Data":"b85da5f71391bb8c8ae86be5aeb1ff4ec5309b4740fdf61c16085dd5286911c8"}
Nov 03 16:19:55.497864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:55.497812    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn" podStartSLOduration=1.895577881 podStartE2EDuration="2.497798511s" totalImagesPullingTime="602.22063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:55.496313783 +0000 UTC m=+557.075617649" watchObservedRunningTime="2026-11-03 16:19:55.497798511 +0000 UTC m=+557.077102365"
Nov 03 16:19:56.298503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:19:56.298470    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:20:02.105929 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.105892    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"]
Nov 03 16:20:02.109738 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.109715    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.167130 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.167100    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"]
Nov 03 16:20:02.205654 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.205623    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.205804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.205667    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.205804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.205729    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/66361ba9-7dcb-49a9-b539-d6a40a5be289-kube-api-access-gplkx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.205924 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.205807    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.205924 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.205850    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66361ba9-7dcb-49a9-b539-d6a40a5be289-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.205924 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.205885    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.206908 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.206892    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 16:20:02.306748 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.306718    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.306748 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.306753    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.306986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.306771    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/66361ba9-7dcb-49a9-b539-d6a40a5be289-kube-api-access-gplkx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.306986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.306899    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.306986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.306949    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66361ba9-7dcb-49a9-b539-d6a40a5be289-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.306986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.306976    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.307255 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.307154    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.307255 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.307240    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.307549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.307508    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.309570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.309547    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66361ba9-7dcb-49a9-b539-d6a40a5be289-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.309730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.309711    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66361ba9-7dcb-49a9-b539-d6a40a5be289-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.362873 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.362813    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/66361ba9-7dcb-49a9-b539-d6a40a5be289-kube-api-access-gplkx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw\" (UID: \"66361ba9-7dcb-49a9-b539-d6a40a5be289\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.423587 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.423562    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:02.797758 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:02.797731    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"]
Nov 03 16:20:02.800033 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:20:02.800004    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66361ba9_7dcb_49a9_b539_d6a40a5be289.slice/crio-cda2a844ff8198814d4434b6319c794ff0b9bf0b0b5a52585966ffd6d013e411 WatchSource:0}: Error finding container cda2a844ff8198814d4434b6319c794ff0b9bf0b0b5a52585966ffd6d013e411: Status 404 returned error can't find the container with id cda2a844ff8198814d4434b6319c794ff0b9bf0b0b5a52585966ffd6d013e411
Nov 03 16:20:03.427149 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:03.427110    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw" event={"ID":"66361ba9-7dcb-49a9-b539-d6a40a5be289","Type":"ContainerStarted","Data":"cda2a844ff8198814d4434b6319c794ff0b9bf0b0b5a52585966ffd6d013e411"}
Nov 03 16:20:04.298991 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:04.298945    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:20:04.311675 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:04.311648    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:20:04.433617 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:04.433576    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw" event={"ID":"66361ba9-7dcb-49a9-b539-d6a40a5be289","Type":"ContainerStarted","Data":"9a445d605e4c6731f88f39cf0e2e9b125a431bc8faa9cfe6581f3a380139e3aa"}
Nov 03 16:20:04.445083 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:04.445060    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn"
Nov 03 16:20:04.548023 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:04.547974    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw" podStartSLOduration=1.8462815799999999 podStartE2EDuration="2.547959433s" totalImagesPullingTime="701.677853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:04.546917451 +0000 UTC m=+566.126221309" watchObservedRunningTime="2026-11-03 16:20:04.547959433 +0000 UTC m=+566.127263328"
Nov 03 16:20:05.424547 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:05.424517    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:07.882811 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:20:07.882771    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:20:12.424526 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:12.424488    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:12.437590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:12.437566    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:12.489149 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:12.489117    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw"
Nov 03 16:20:29.007676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.007637    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66ff8f5b47-kph6d"]
Nov 03 16:20:29.011423 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.011396    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.053809 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.053783    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f4c81cc-7834-49dd-b89d-5b4b3189e013-tls-cert\") pod \"authorino-66ff8f5b47-kph6d\" (UID: \"3f4c81cc-7834-49dd-b89d-5b4b3189e013\") " pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.053930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.053815    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btzd\" (UniqueName: \"kubernetes.io/projected/3f4c81cc-7834-49dd-b89d-5b4b3189e013-kube-api-access-5btzd\") pod \"authorino-66ff8f5b47-kph6d\" (UID: \"3f4c81cc-7834-49dd-b89d-5b4b3189e013\") " pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.067921 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.067887    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66ff8f5b47-kph6d"]
Nov 03 16:20:29.112778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.112749    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 16:20:29.112778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.112759    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zd89p\""
Nov 03 16:20:29.155034 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.155007    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f4c81cc-7834-49dd-b89d-5b4b3189e013-tls-cert\") pod \"authorino-66ff8f5b47-kph6d\" (UID: \"3f4c81cc-7834-49dd-b89d-5b4b3189e013\") " pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.155034 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.155037    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btzd\" (UniqueName: \"kubernetes.io/projected/3f4c81cc-7834-49dd-b89d-5b4b3189e013-kube-api-access-5btzd\") pod \"authorino-66ff8f5b47-kph6d\" (UID: \"3f4c81cc-7834-49dd-b89d-5b4b3189e013\") " pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.157564 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.157533    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f4c81cc-7834-49dd-b89d-5b4b3189e013-tls-cert\") pod \"authorino-66ff8f5b47-kph6d\" (UID: \"3f4c81cc-7834-49dd-b89d-5b4b3189e013\") " pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.167453 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.167427    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-w9zhv"]
Nov 03 16:20:29.170963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.170943    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:29.212066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.212023    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btzd\" (UniqueName: \"kubernetes.io/projected/3f4c81cc-7834-49dd-b89d-5b4b3189e013-kube-api-access-5btzd\") pod \"authorino-66ff8f5b47-kph6d\" (UID: \"3f4c81cc-7834-49dd-b89d-5b4b3189e013\") " pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.231474 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.231439    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-w9zhv"]
Nov 03 16:20:29.239451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.239423    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 16:20:29.239451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.239427    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 16:20:29.239698 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.239438    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-89v2n\""
Nov 03 16:20:29.256163 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.256128    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs\") pod \"ai-gateway-controller-79567948bc-w9zhv\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") " pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:29.256318 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.256306    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2gj\" (UniqueName: \"kubernetes.io/projected/ab61dff5-996c-4850-8823-79f07c869cd7-kube-api-access-qb2gj\") pod \"ai-gateway-controller-79567948bc-w9zhv\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") " pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:29.323456 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.323374    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66ff8f5b47-kph6d"
Nov 03 16:20:29.357398 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.357366    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2gj\" (UniqueName: \"kubernetes.io/projected/ab61dff5-996c-4850-8823-79f07c869cd7-kube-api-access-qb2gj\") pod \"ai-gateway-controller-79567948bc-w9zhv\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") " pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:29.357570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.357453    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs\") pod \"ai-gateway-controller-79567948bc-w9zhv\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") " pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:29.357570 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:20:29.357560    5112 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found
Nov 03 16:20:29.357682 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:20:29.357621    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs podName:ab61dff5-996c-4850-8823-79f07c869cd7 nodeName:}" failed. No retries permitted until 2026-11-03 16:20:29.857605044 +0000 UTC m=+591.436908877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs") pod "ai-gateway-controller-79567948bc-w9zhv" (UID: "ab61dff5-996c-4850-8823-79f07c869cd7") : secret "ai-gateway-controller-metrics-tls" not found
Nov 03 16:20:29.415033 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.414999    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2gj\" (UniqueName: \"kubernetes.io/projected/ab61dff5-996c-4850-8823-79f07c869cd7-kube-api-access-qb2gj\") pod \"ai-gateway-controller-79567948bc-w9zhv\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") " pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:29.693189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.693162    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66ff8f5b47-kph6d"]
Nov 03 16:20:29.695191 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:20:29.695161    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f4c81cc_7834_49dd_b89d_5b4b3189e013.slice/crio-d6671014da7b79719daba93ec1225b1611557a0aec9fe41c55565bc8fbad18b0 WatchSource:0}: Error finding container d6671014da7b79719daba93ec1225b1611557a0aec9fe41c55565bc8fbad18b0: Status 404 returned error can't find the container with id d6671014da7b79719daba93ec1225b1611557a0aec9fe41c55565bc8fbad18b0
Nov 03 16:20:29.862621 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.862583    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs\") pod \"ai-gateway-controller-79567948bc-w9zhv\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") " pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:29.865036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:29.865013    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs\") pod \"ai-gateway-controller-79567948bc-w9zhv\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") " pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:30.084797 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:30.084693    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:30.458110 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:30.458088    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-w9zhv"]
Nov 03 16:20:30.460428 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:20:30.460391    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab61dff5_996c_4850_8823_79f07c869cd7.slice/crio-6dcac623f19776e93bef394bfc1558a0141b9e32cfc64c6617e3dba7c525fb97 WatchSource:0}: Error finding container 6dcac623f19776e93bef394bfc1558a0141b9e32cfc64c6617e3dba7c525fb97: Status 404 returned error can't find the container with id 6dcac623f19776e93bef394bfc1558a0141b9e32cfc64c6617e3dba7c525fb97
Nov 03 16:20:30.577392 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:30.577359    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv" event={"ID":"ab61dff5-996c-4850-8823-79f07c869cd7","Type":"ContainerStarted","Data":"6dcac623f19776e93bef394bfc1558a0141b9e32cfc64c6617e3dba7c525fb97"}
Nov 03 16:20:30.578575 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:30.578550    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66ff8f5b47-kph6d" event={"ID":"3f4c81cc-7834-49dd-b89d-5b4b3189e013","Type":"ContainerStarted","Data":"d6671014da7b79719daba93ec1225b1611557a0aec9fe41c55565bc8fbad18b0"}
Nov 03 16:20:31.585054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:31.585019    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66ff8f5b47-kph6d" event={"ID":"3f4c81cc-7834-49dd-b89d-5b4b3189e013","Type":"ContainerStarted","Data":"e65b07ac1193bf305e4d193c5966f0d37be6edd9110a48ec62b250ecd1fa443a"}
Nov 03 16:20:31.702080 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:31.701995    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66ff8f5b47-kph6d" podStartSLOduration=1.991272341 podStartE2EDuration="3.701981452s" totalImagesPullingTime="1.710709111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:31.699954528 +0000 UTC m=+593.279258408" watchObservedRunningTime="2026-11-03 16:20:31.701981452 +0000 UTC m=+593.281285351"
Nov 03 16:20:37.631145 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:37.631105    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv" event={"ID":"ab61dff5-996c-4850-8823-79f07c869cd7","Type":"ContainerStarted","Data":"1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b"}
Nov 03 16:20:37.631700 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:37.631243    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:37.746921 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:37.746869    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv" podStartSLOduration=2.6372602240000003 podStartE2EDuration="8.746855058s" totalImagesPullingTime="6.109594834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:37.745941985 +0000 UTC m=+599.325245839" watchObservedRunningTime="2026-11-03 16:20:37.746855058 +0000 UTC m=+599.326158913"
Nov 03 16:20:37.887189 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:20:37.887091    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:20:38.600853 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:38.600825    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:20:38.601039 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:38.600830    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:20:38.611682 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:38.611661    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:20:38.611851 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:38.611835    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:20:39.744155 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.744118    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"]
Nov 03 16:20:39.748155 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.748135    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:39.804184 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.804148    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"]
Nov 03 16:20:39.849145 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.849111    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 16:20:39.849145 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.849112    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vr8mb\""
Nov 03 16:20:39.851413 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.851386    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"]
Nov 03 16:20:39.855345 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.855325    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:39.860110 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.860082    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b88bbf9-f49b-4e54-9eea-900cc0e5e078-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-b9ppx\" (UID: \"9b88bbf9-f49b-4e54-9eea-900cc0e5e078\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:39.860254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.860130    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69np\" (UniqueName: \"kubernetes.io/projected/9b88bbf9-f49b-4e54-9eea-900cc0e5e078-kube-api-access-v69np\") pod \"payload-pre-processing-7b98bb678c-b9ppx\" (UID: \"9b88bbf9-f49b-4e54-9eea-900cc0e5e078\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:39.910048 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.910016    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"]
Nov 03 16:20:39.961595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.961561    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69np\" (UniqueName: \"kubernetes.io/projected/9b88bbf9-f49b-4e54-9eea-900cc0e5e078-kube-api-access-v69np\") pod \"payload-pre-processing-7b98bb678c-b9ppx\" (UID: \"9b88bbf9-f49b-4e54-9eea-900cc0e5e078\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:39.961751 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.961673    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmtd\" (UniqueName: \"kubernetes.io/projected/8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4-kube-api-access-vzmtd\") pod \"payload-processing-55fbbbf9dc-5j5sw\" (UID: \"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:39.961751 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.961694    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5j5sw\" (UID: \"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:39.961751 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.961720    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b88bbf9-f49b-4e54-9eea-900cc0e5e078-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-b9ppx\" (UID: \"9b88bbf9-f49b-4e54-9eea-900cc0e5e078\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:39.962483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:39.962464    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b88bbf9-f49b-4e54-9eea-900cc0e5e078-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-b9ppx\" (UID: \"9b88bbf9-f49b-4e54-9eea-900cc0e5e078\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:40.018503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.018425    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69np\" (UniqueName: \"kubernetes.io/projected/9b88bbf9-f49b-4e54-9eea-900cc0e5e078-kube-api-access-v69np\") pod \"payload-pre-processing-7b98bb678c-b9ppx\" (UID: \"9b88bbf9-f49b-4e54-9eea-900cc0e5e078\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:40.061458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.061420    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:40.062797 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.062773    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmtd\" (UniqueName: \"kubernetes.io/projected/8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4-kube-api-access-vzmtd\") pod \"payload-processing-55fbbbf9dc-5j5sw\" (UID: \"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:40.062896 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.062810    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5j5sw\" (UID: \"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:40.063529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.063510    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-5j5sw\" (UID: \"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:40.121390 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.121362    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmtd\" (UniqueName: \"kubernetes.io/projected/8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4-kube-api-access-vzmtd\") pod \"payload-processing-55fbbbf9dc-5j5sw\" (UID: \"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:40.169711 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.169668    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:40.458401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.458376    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"]
Nov 03 16:20:40.459974 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:20:40.459946    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b88bbf9_f49b_4e54_9eea_900cc0e5e078.slice/crio-9ac77b86d6936e796d9d4b0d1a2f499d79724f967f9b8b469a06a0e5bbcb80ef WatchSource:0}: Error finding container 9ac77b86d6936e796d9d4b0d1a2f499d79724f967f9b8b469a06a0e5bbcb80ef: Status 404 returned error can't find the container with id 9ac77b86d6936e796d9d4b0d1a2f499d79724f967f9b8b469a06a0e5bbcb80ef
Nov 03 16:20:40.588796 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.588763    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"]
Nov 03 16:20:40.590370 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:20:40.590336    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1a9cc3_468f_45fc_a0b9_bf9bd6ecc7d4.slice/crio-61215528bf3486211867239ae9b6eadd1b8301c79ac3ccc9c1ac9fa096ebb138 WatchSource:0}: Error finding container 61215528bf3486211867239ae9b6eadd1b8301c79ac3ccc9c1ac9fa096ebb138: Status 404 returned error can't find the container with id 61215528bf3486211867239ae9b6eadd1b8301c79ac3ccc9c1ac9fa096ebb138
Nov 03 16:20:40.650597 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.650553    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx" event={"ID":"9b88bbf9-f49b-4e54-9eea-900cc0e5e078","Type":"ContainerStarted","Data":"9ac77b86d6936e796d9d4b0d1a2f499d79724f967f9b8b469a06a0e5bbcb80ef"}
Nov 03 16:20:40.651750 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:40.651721    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw" event={"ID":"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4","Type":"ContainerStarted","Data":"61215528bf3486211867239ae9b6eadd1b8301c79ac3ccc9c1ac9fa096ebb138"}
Nov 03 16:20:44.676850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:44.676812    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx" event={"ID":"9b88bbf9-f49b-4e54-9eea-900cc0e5e078","Type":"ContainerStarted","Data":"35bbfb726983dd888beeaba7c99888e3453fba5a6773759c3f0964050d22a8e5"}
Nov 03 16:20:44.677347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:44.676872    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:20:44.678277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:44.678254    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw" event={"ID":"8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4","Type":"ContainerStarted","Data":"9ba43c4495a25dcd24a63f9222d3fa89ec0c518c46d6386cf6b04fbe99e12159"}
Nov 03 16:20:44.678376 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:44.678292    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:44.794662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:44.794614    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx" podStartSLOduration=2.379598823 podStartE2EDuration="5.794600121s" totalImagesPullingTime="3.415001298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:44.791971974 +0000 UTC m=+606.371275826" watchObservedRunningTime="2026-11-03 16:20:44.794600121 +0000 UTC m=+606.373903973"
Nov 03 16:20:44.910565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:44.910513    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw" podStartSLOduration=2.330603001 podStartE2EDuration="5.910447629s" totalImagesPullingTime="3.579844628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:44.910369199 +0000 UTC m=+606.489673074" watchObservedRunningTime="2026-11-03 16:20:44.910447629 +0000 UTC m=+606.489751481"
Nov 03 16:20:48.639857 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:48.639830    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:20:48.993814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:48.993770    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-778b77c46d-wg5sn"]
Nov 03 16:20:48.997723 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:48.997702    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.053792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.053761    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-778b77c46d-wg5sn"]
Nov 03 16:20:49.149380 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.149343    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f58454f-bc25-4bae-a8ff-aa43c025d537-metrics-certs\") pod \"ai-gateway-controller-778b77c46d-wg5sn\" (UID: \"3f58454f-bc25-4bae-a8ff-aa43c025d537\") " pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.149542 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.149463    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkmxj\" (UniqueName: \"kubernetes.io/projected/3f58454f-bc25-4bae-a8ff-aa43c025d537-kube-api-access-nkmxj\") pod \"ai-gateway-controller-778b77c46d-wg5sn\" (UID: \"3f58454f-bc25-4bae-a8ff-aa43c025d537\") " pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.250073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.249996    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkmxj\" (UniqueName: \"kubernetes.io/projected/3f58454f-bc25-4bae-a8ff-aa43c025d537-kube-api-access-nkmxj\") pod \"ai-gateway-controller-778b77c46d-wg5sn\" (UID: \"3f58454f-bc25-4bae-a8ff-aa43c025d537\") " pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.250073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.250064    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f58454f-bc25-4bae-a8ff-aa43c025d537-metrics-certs\") pod \"ai-gateway-controller-778b77c46d-wg5sn\" (UID: \"3f58454f-bc25-4bae-a8ff-aa43c025d537\") " pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.252467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.252441    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f58454f-bc25-4bae-a8ff-aa43c025d537-metrics-certs\") pod \"ai-gateway-controller-778b77c46d-wg5sn\" (UID: \"3f58454f-bc25-4bae-a8ff-aa43c025d537\") " pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.307044 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.307015    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkmxj\" (UniqueName: \"kubernetes.io/projected/3f58454f-bc25-4bae-a8ff-aa43c025d537-kube-api-access-nkmxj\") pod \"ai-gateway-controller-778b77c46d-wg5sn\" (UID: \"3f58454f-bc25-4bae-a8ff-aa43c025d537\") " pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.310712 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.310693    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:49.711849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.711816    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-778b77c46d-wg5sn"]
Nov 03 16:20:49.717802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:49.717772    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:20:50.711521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:50.711480    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn" event={"ID":"3f58454f-bc25-4bae-a8ff-aa43c025d537","Type":"ContainerStarted","Data":"ce94202104d868ccbe2de8f7dad095ce602ea5aafa179038755796f1094961d2"}
Nov 03 16:20:50.711676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:50.711545    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn" event={"ID":"3f58454f-bc25-4bae-a8ff-aa43c025d537","Type":"ContainerStarted","Data":"428e9215826049e4fe00dcba623436fccfac1da9efc89af4e3d7edd737a49115"}
Nov 03 16:20:50.711676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:50.711580    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:20:50.825786 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:50.825736    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn" podStartSLOduration=1.946205999 podStartE2EDuration="2.825722979s" totalImagesPullingTime="879.51698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:50.824562159 +0000 UTC m=+612.403866044" watchObservedRunningTime="2026-11-03 16:20:50.825722979 +0000 UTC m=+612.405026861"
Nov 03 16:20:55.687501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:55.687465    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-5j5sw"
Nov 03 16:20:55.687887 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:20:55.687534    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx"
Nov 03 16:21:01.719501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:01.719471    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-778b77c46d-wg5sn"
Nov 03 16:21:01.883671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:01.883634    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-w9zhv"]
Nov 03 16:21:01.883935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:01.883903    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv" podUID="ab61dff5-996c-4850-8823-79f07c869cd7" containerName="manager" containerID="cri-o://1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b" gracePeriod=30
Nov 03 16:21:02.154422 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.154393    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:21:02.271148 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.271060    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs\") pod \"ab61dff5-996c-4850-8823-79f07c869cd7\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") "
Nov 03 16:21:02.271346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.271154    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab61dff5-996c-4850-8823-79f07c869cd7-kube-api-access-qb2gj\" (UniqueName: \"kubernetes.io/projected/ab61dff5-996c-4850-8823-79f07c869cd7-kube-api-access-qb2gj\") pod \"ab61dff5-996c-4850-8823-79f07c869cd7\" (UID: \"ab61dff5-996c-4850-8823-79f07c869cd7\") "
Nov 03 16:21:02.274115 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.274071    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs" pod "ab61dff5-996c-4850-8823-79f07c869cd7" (UID: "ab61dff5-996c-4850-8823-79f07c869cd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:21:02.275082 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.275045    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab61dff5-996c-4850-8823-79f07c869cd7-kube-api-access-qb2gj" pod "ab61dff5-996c-4850-8823-79f07c869cd7" (UID: "ab61dff5-996c-4850-8823-79f07c869cd7"). InnerVolumeSpecName "kube-api-access-qb2gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:21:02.372319 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.372284    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab61dff5-996c-4850-8823-79f07c869cd7-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:02.372319 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.372315    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2gj\" (UniqueName: \"kubernetes.io/projected/ab61dff5-996c-4850-8823-79f07c869cd7-kube-api-access-qb2gj\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:21:02.776574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.776539    5112 generic.go:356] "Generic (PLEG): container finished" podID="ab61dff5-996c-4850-8823-79f07c869cd7" containerID="1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b" exitCode=0
Nov 03 16:21:02.776990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.776587    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv"
Nov 03 16:21:02.776990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.776605    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv" event={"ID":"ab61dff5-996c-4850-8823-79f07c869cd7","Type":"ContainerDied","Data":"1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b"}
Nov 03 16:21:02.776990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.776647    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79567948bc-w9zhv" event={"ID":"ab61dff5-996c-4850-8823-79f07c869cd7","Type":"ContainerDied","Data":"6dcac623f19776e93bef394bfc1558a0141b9e32cfc64c6617e3dba7c525fb97"}
Nov 03 16:21:02.776990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.776667    5112 scope.go:122] "RemoveContainer" containerID="1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b"
Nov 03 16:21:02.787937 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.787911    5112 scope.go:122] "RemoveContainer" containerID="1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b"
Nov 03 16:21:02.788350 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:21:02.788324    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b\": container with ID starting with 1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b not found: ID does not exist" containerID="1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b"
Nov 03 16:21:02.788458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.788359    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b"} err="failed to get container status \"1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b\": rpc error: code = NotFound desc = could not find container \"1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b\": container with ID starting with 1b680fde53ef040d0c773bf82d5931646a5ad69f5bd6cfec4feab12e8edb499b not found: ID does not exist"
Nov 03 16:21:02.939973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.939940    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-w9zhv"]
Nov 03 16:21:02.986935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:02.986889    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-79567948bc-w9zhv"]
Nov 03 16:21:04.548597 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:21:04.548563    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab61dff5-996c-4850-8823-79f07c869cd7" path="/var/lib/kubelet/pods/ab61dff5-996c-4850-8823-79f07c869cd7/volumes"
Nov 03 16:21:07.881652 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:21:07.881605    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:21:37.883640 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:21:37.883597    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:22:04.448036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.448000    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"]
Nov 03 16:22:04.448568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.448505    5112 cpu_manager.go:414] "containerMap: removing container" podUID="ab61dff5-996c-4850-8823-79f07c869cd7" containerName="manager"
Nov 03 16:22:04.448568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.448517    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab61dff5-996c-4850-8823-79f07c869cd7" containerName="manager"
Nov 03 16:22:04.448652 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.448615    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab61dff5-996c-4850-8823-79f07c869cd7" containerName="manager"
Nov 03 16:22:04.452065 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.452042    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.475743 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.475701    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.475899 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.475771    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsd8q\" (UniqueName: \"kubernetes.io/projected/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-kube-api-access-qsd8q\") pod \"payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.563512 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.563480    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-42da0f-dockercfg-fg7b8\""
Nov 03 16:22:04.564409 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.564356    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-42da0f\""
Nov 03 16:22:04.566295 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.566272    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"]
Nov 03 16:22:04.576466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.576438    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsd8q\" (UniqueName: \"kubernetes.io/projected/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-kube-api-access-qsd8q\") pod \"payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.576629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.576561    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.577385 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.577360    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.606882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.606849    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"]
Nov 03 16:22:04.610763 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.610746    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:04.632813 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.632784    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsd8q\" (UniqueName: \"kubernetes.io/projected/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-kube-api-access-qsd8q\") pod \"payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.647325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.647297    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"]
Nov 03 16:22:04.677260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.677187    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ff4aac-032d-483d-8ea3-648619a23e2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:04.677447 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.677371    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp7nx\" (UniqueName: \"kubernetes.io/projected/b0ff4aac-032d-483d-8ea3-648619a23e2d-kube-api-access-xp7nx\") pod \"payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:04.766302 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.766174    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:04.778710 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.778679    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ff4aac-032d-483d-8ea3-648619a23e2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:04.778883 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.778785    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp7nx\" (UniqueName: \"kubernetes.io/projected/b0ff4aac-032d-483d-8ea3-648619a23e2d-kube-api-access-xp7nx\") pod \"payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:04.779686 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.779658    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ff4aac-032d-483d-8ea3-648619a23e2d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:04.849334 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.849256    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp7nx\" (UniqueName: \"kubernetes.io/projected/b0ff4aac-032d-483d-8ea3-648619a23e2d-kube-api-access-xp7nx\") pod \"payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:04.924998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:04.924960    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:05.145957 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.145925    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"]
Nov 03 16:22:05.147972 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:05.147933    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105b3420_4eb9_4bb0_bdc3_afb6bee8f9eb.slice/crio-bcd8a7cedb988fd8602987f4c8ee85562900da0b463b50b1f2bb2141f31373a4 WatchSource:0}: Error finding container bcd8a7cedb988fd8602987f4c8ee85562900da0b463b50b1f2bb2141f31373a4: Status 404 returned error can't find the container with id bcd8a7cedb988fd8602987f4c8ee85562900da0b463b50b1f2bb2141f31373a4
Nov 03 16:22:05.214870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.214844    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"]
Nov 03 16:22:05.218650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.218629    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.277958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.277930    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"]
Nov 03 16:22:05.283070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.283037    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.283245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.283160    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-metrics-certs\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.283329 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.283257    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdlq\" (UniqueName: \"kubernetes.io/projected/8fc2a681-7950-4b44-89d2-18f34e8ae930-kube-api-access-6vdlq\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.283393 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.283329    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2a681-7950-4b44-89d2-18f34e8ae930-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.306819 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:05.306779    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ff4aac_032d_483d_8ea3_648619a23e2d.slice/crio-52ad250d6c0e5ff53ee92a03d75c76feb5ec4f50cc753ff810c2cef83d8a71a9 WatchSource:0}: Error finding container 52ad250d6c0e5ff53ee92a03d75c76feb5ec4f50cc753ff810c2cef83d8a71a9: Status 404 returned error can't find the container with id 52ad250d6c0e5ff53ee92a03d75c76feb5ec4f50cc753ff810c2cef83d8a71a9
Nov 03 16:22:05.318811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.318783    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-6ae690-serving-cert\""
Nov 03 16:22:05.318811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.318801    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7k42d\""
Nov 03 16:22:05.318811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.318786    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 16:22:05.321216 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.321157    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"]
Nov 03 16:22:05.383994 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.383956    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-metrics-certs\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.384160 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.384032    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdlq\" (UniqueName: \"kubernetes.io/projected/8fc2a681-7950-4b44-89d2-18f34e8ae930-kube-api-access-6vdlq\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.384160 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.384098    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2a681-7950-4b44-89d2-18f34e8ae930-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.384290 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.384238    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.384412 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:05.384393    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-serving-cert: secret "maas-api-e2e-worker-w5-6ae690-serving-cert" not found
Nov 03 16:22:05.384485 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:05.384471    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls podName:8fc2a681-7950-4b44-89d2-18f34e8ae930 nodeName:}" failed. No retries permitted until 2026-11-03 16:22:05.884450119 +0000 UTC m=+687.463753970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls") pod "maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2" (UID: "8fc2a681-7950-4b44-89d2-18f34e8ae930") : secret "maas-api-e2e-worker-w5-6ae690-serving-cert" not found
Nov 03 16:22:05.384796 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.384777    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2a681-7950-4b44-89d2-18f34e8ae930-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.386430 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.386412    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-metrics-certs\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.441699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.441612    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdlq\" (UniqueName: \"kubernetes.io/projected/8fc2a681-7950-4b44-89d2-18f34e8ae930-kube-api-access-6vdlq\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.888086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.887995    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:05.890529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:05.890496    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:06.114114 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.114074    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w" event={"ID":"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb","Type":"ContainerStarted","Data":"6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9"}
Nov 03 16:22:06.114397 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.114142    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w" event={"ID":"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb","Type":"ContainerStarted","Data":"bcd8a7cedb988fd8602987f4c8ee85562900da0b463b50b1f2bb2141f31373a4"}
Nov 03 16:22:06.114397 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.114170    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:06.115794 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.115768    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv" event={"ID":"b0ff4aac-032d-483d-8ea3-648619a23e2d","Type":"ContainerStarted","Data":"ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b"}
Nov 03 16:22:06.115902 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.115813    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv" event={"ID":"b0ff4aac-032d-483d-8ea3-648619a23e2d","Type":"ContainerStarted","Data":"52ad250d6c0e5ff53ee92a03d75c76feb5ec4f50cc753ff810c2cef83d8a71a9"}
Nov 03 16:22:06.115902 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.115843    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:06.132242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.132183    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:06.227562 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.227505    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w" podStartSLOduration=2.22749079 podStartE2EDuration="2.22749079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:06.225516621 +0000 UTC m=+687.804820496" watchObservedRunningTime="2026-11-03 16:22:06.22749079 +0000 UTC m=+687.806794645"
Nov 03 16:22:06.336812 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.336752    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv" podStartSLOduration=2.336737952 podStartE2EDuration="2.336737952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:06.335999962 +0000 UTC m=+687.915303817" watchObservedRunningTime="2026-11-03 16:22:06.336737952 +0000 UTC m=+687.916041806"
Nov 03 16:22:06.558979 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:06.558947    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"]
Nov 03 16:22:06.561363 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:06.561330    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc2a681_7950_4b44_89d2_18f34e8ae930.slice/crio-f27ace0ed2c297dccdc928da03d2311498cdacfbc30afaffdaafc8ea5e7309eb WatchSource:0}: Error finding container f27ace0ed2c297dccdc928da03d2311498cdacfbc30afaffdaafc8ea5e7309eb: Status 404 returned error can't find the container with id f27ace0ed2c297dccdc928da03d2311498cdacfbc30afaffdaafc8ea5e7309eb
Nov 03 16:22:07.122963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:07.122917    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2" event={"ID":"8fc2a681-7950-4b44-89d2-18f34e8ae930","Type":"ContainerStarted","Data":"f27ace0ed2c297dccdc928da03d2311498cdacfbc30afaffdaafc8ea5e7309eb"}
Nov 03 16:22:07.883673 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:07.883581    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:22:08.129814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:08.129779    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2" event={"ID":"8fc2a681-7950-4b44-89d2-18f34e8ae930","Type":"ContainerStarted","Data":"3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb"}
Nov 03 16:22:08.130275 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:08.129936    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:08.245699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:08.245649    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2" podStartSLOduration=2.27057061 podStartE2EDuration="3.245635715s" totalImagesPullingTime="975.065105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:08.245088855 +0000 UTC m=+689.824392728" watchObservedRunningTime="2026-11-03 16:22:08.245635715 +0000 UTC m=+689.824939570"
Nov 03 16:22:09.532488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.532432    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"]
Nov 03 16:22:09.536428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.536409    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.593572 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.593538    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"]
Nov 03 16:22:09.626141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.626100    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9t8\" (UniqueName: \"kubernetes.io/projected/b08b03b9-4124-4294-bab1-29e4c4c85917-kube-api-access-rq9t8\") pod \"payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.626646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.626619    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08b03b9-4124-4294-bab1-29e4c4c85917-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.636930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.636909    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6ae690\""
Nov 03 16:22:09.636930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.636915    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6ae690-dockercfg-zgk8t\""
Nov 03 16:22:09.687679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.687645    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"]
Nov 03 16:22:09.692018 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.691992    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:09.727545 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.727510    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9t8\" (UniqueName: \"kubernetes.io/projected/b08b03b9-4124-4294-bab1-29e4c4c85917-kube-api-access-rq9t8\") pod \"payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.727721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.727607    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08b03b9-4124-4294-bab1-29e4c4c85917-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.727721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.727651    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b706ed-062e-40a3-8044-bafc0c131f07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:09.727721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.727700    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmrt\" (UniqueName: \"kubernetes.io/projected/88b706ed-062e-40a3-8044-bafc0c131f07-kube-api-access-2nmrt\") pod \"payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:09.728354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.728336    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08b03b9-4124-4294-bab1-29e4c4c85917-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.744230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.744177    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"]
Nov 03 16:22:09.785574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.785493    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9t8\" (UniqueName: \"kubernetes.io/projected/b08b03b9-4124-4294-bab1-29e4c4c85917-kube-api-access-rq9t8\") pod \"payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.828737 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.828707    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b706ed-062e-40a3-8044-bafc0c131f07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:09.828904 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.828755    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmrt\" (UniqueName: \"kubernetes.io/projected/88b706ed-062e-40a3-8044-bafc0c131f07-kube-api-access-2nmrt\") pod \"payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:09.829475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.829456    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b706ed-062e-40a3-8044-bafc0c131f07-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:09.850322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.850294    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:09.886817 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:09.886778    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmrt\" (UniqueName: \"kubernetes.io/projected/88b706ed-062e-40a3-8044-bafc0c131f07-kube-api-access-2nmrt\") pod \"payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:10.007469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.007430    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:10.228358 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.228332    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"]
Nov 03 16:22:10.229959 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:10.229937    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08b03b9_4124_4294_bab1_29e4c4c85917.slice/crio-09507d84a5b71315c0c6880e4860e216a474e1524dc01a899d788cdf4b5abb6c WatchSource:0}: Error finding container 09507d84a5b71315c0c6880e4860e216a474e1524dc01a899d788cdf4b5abb6c: Status 404 returned error can't find the container with id 09507d84a5b71315c0c6880e4860e216a474e1524dc01a899d788cdf4b5abb6c
Nov 03 16:22:10.380776 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.380746    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"]
Nov 03 16:22:10.383035 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:10.383006    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b706ed_062e_40a3_8044_bafc0c131f07.slice/crio-a31ba38b17988e95c826e0440ba59b7fe61aa04cb239aa0a55107e82592d276d WatchSource:0}: Error finding container a31ba38b17988e95c826e0440ba59b7fe61aa04cb239aa0a55107e82592d276d: Status 404 returned error can't find the container with id a31ba38b17988e95c826e0440ba59b7fe61aa04cb239aa0a55107e82592d276d
Nov 03 16:22:10.400439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.400415    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"]
Nov 03 16:22:10.405494 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.405474    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.435023 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.434993    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-metrics-certs\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.435158 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.435042    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.436501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.436471    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vlmq\" (UniqueName: \"kubernetes.io/projected/9cc4c08f-e98d-49ed-a69e-41fc144d0273-kube-api-access-2vlmq\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.436622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.436519    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc4c08f-e98d-49ed-a69e-41fc144d0273-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.459702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.459679    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-c88d54-serving-cert\""
Nov 03 16:22:10.461852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.461824    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"]
Nov 03 16:22:10.537807 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.537768    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-metrics-certs\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.537807 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.537812    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.538402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.537896    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vlmq\" (UniqueName: \"kubernetes.io/projected/9cc4c08f-e98d-49ed-a69e-41fc144d0273-kube-api-access-2vlmq\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.538402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.537917    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc4c08f-e98d-49ed-a69e-41fc144d0273-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.538402 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:10.537943    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-serving-cert: secret "maas-api-e2e-worker-w2-c88d54-serving-cert" not found
Nov 03 16:22:10.538402 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:10.538013    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls podName:9cc4c08f-e98d-49ed-a69e-41fc144d0273 nodeName:}" failed. No retries permitted until 2026-11-03 16:22:11.037994702 +0000 UTC m=+692.617298555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls") pod "maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp" (UID: "9cc4c08f-e98d-49ed-a69e-41fc144d0273") : secret "maas-api-e2e-worker-w2-c88d54-serving-cert" not found
Nov 03 16:22:10.538615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.538597    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc4c08f-e98d-49ed-a69e-41fc144d0273-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.540469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.540448    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-metrics-certs\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:10.595503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:10.595465    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vlmq\" (UniqueName: \"kubernetes.io/projected/9cc4c08f-e98d-49ed-a69e-41fc144d0273-kube-api-access-2vlmq\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:11.043993 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.043954    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:11.046475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.046448    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:11.152995 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.152956    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" event={"ID":"88b706ed-062e-40a3-8044-bafc0c131f07","Type":"ContainerStarted","Data":"c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28"}
Nov 03 16:22:11.153238 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.153017    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" event={"ID":"88b706ed-062e-40a3-8044-bafc0c131f07","Type":"ContainerStarted","Data":"a31ba38b17988e95c826e0440ba59b7fe61aa04cb239aa0a55107e82592d276d"}
Nov 03 16:22:11.153238 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.153048    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:11.154564 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.154540    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" event={"ID":"b08b03b9-4124-4294-bab1-29e4c4c85917","Type":"ContainerStarted","Data":"66dccaa33192035c1168e82b9d5f0761f96af7fe1cc76218db395a2831c0b84c"}
Nov 03 16:22:11.154674 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.154580    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" event={"ID":"b08b03b9-4124-4294-bab1-29e4c4c85917","Type":"ContainerStarted","Data":"09507d84a5b71315c0c6880e4860e216a474e1524dc01a899d788cdf4b5abb6c"}
Nov 03 16:22:11.154729 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.154686    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:11.265257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.265168    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" podStartSLOduration=2.265150176 podStartE2EDuration="2.265150176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:11.264483315 +0000 UTC m=+692.843787190" watchObservedRunningTime="2026-11-03 16:22:11.265150176 +0000 UTC m=+692.844454032"
Nov 03 16:22:11.344102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.344009    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:11.381347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.381292    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" podStartSLOduration=2.381275704 podStartE2EDuration="2.381275704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:11.379783829 +0000 UTC m=+692.959087713" watchObservedRunningTime="2026-11-03 16:22:11.381275704 +0000 UTC m=+692.960579558"
Nov 03 16:22:11.772316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:11.772286    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"]
Nov 03 16:22:11.774184 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:11.774154    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc4c08f_e98d_49ed_a69e_41fc144d0273.slice/crio-180d53b6d0abac4fd4f28490d36c04f751b5b1705ee1d50b250662765ab5ce1f WatchSource:0}: Error finding container 180d53b6d0abac4fd4f28490d36c04f751b5b1705ee1d50b250662765ab5ce1f: Status 404 returned error can't find the container with id 180d53b6d0abac4fd4f28490d36c04f751b5b1705ee1d50b250662765ab5ce1f
Nov 03 16:22:12.165436 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.165398    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp" event={"ID":"9cc4c08f-e98d-49ed-a69e-41fc144d0273","Type":"ContainerStarted","Data":"180d53b6d0abac4fd4f28490d36c04f751b5b1705ee1d50b250662765ab5ce1f"}
Nov 03 16:22:12.183185 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.183151    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"]
Nov 03 16:22:12.188457 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.188436    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.245618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.245587    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"]
Nov 03 16:22:12.256363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.256334    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcgf\" (UniqueName: \"kubernetes.io/projected/e5229c4c-b2fb-4418-ab4c-89458192001e-kube-api-access-zzcgf\") pod \"payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.256558 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.256394    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5229c4c-b2fb-4418-ab4c-89458192001e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.288600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.288557    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-c88d54\""
Nov 03 16:22:12.288600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.288585    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-c88d54-dockercfg-x6ngw\""
Nov 03 16:22:12.338073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.338036    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"]
Nov 03 16:22:12.342418 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.342396    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.358014 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.357987    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcgf\" (UniqueName: \"kubernetes.io/projected/e5229c4c-b2fb-4418-ab4c-89458192001e-kube-api-access-zzcgf\") pod \"payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.358169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.358041    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5229c4c-b2fb-4418-ab4c-89458192001e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.358724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.358694    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5229c4c-b2fb-4418-ab4c-89458192001e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.381045 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.381021    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"]
Nov 03 16:22:12.414768 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.414733    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcgf\" (UniqueName: \"kubernetes.io/projected/e5229c4c-b2fb-4418-ab4c-89458192001e-kube-api-access-zzcgf\") pod \"payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.458719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.458629    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e4908-9236-4452-a3e1-3f26a5fc0449-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.458900 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.458777    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7p2c\" (UniqueName: \"kubernetes.io/projected/2c6e4908-9236-4452-a3e1-3f26a5fc0449-kube-api-access-h7p2c\") pod \"payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.502772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.502746    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:12.560437 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.560402    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e4908-9236-4452-a3e1-3f26a5fc0449-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.560574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.560520    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7p2c\" (UniqueName: \"kubernetes.io/projected/2c6e4908-9236-4452-a3e1-3f26a5fc0449-kube-api-access-h7p2c\") pod \"payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.561308 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.561286    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e4908-9236-4452-a3e1-3f26a5fc0449-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.618445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.618405    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7p2c\" (UniqueName: \"kubernetes.io/projected/2c6e4908-9236-4452-a3e1-3f26a5fc0449-kube-api-access-h7p2c\") pod \"payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.656363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.656331    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:12.878027 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:12.877999    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"]
Nov 03 16:22:12.879406 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:12.879372    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5229c4c_b2fb_4418_ab4c_89458192001e.slice/crio-6786561f02672e12e4efcba9926772dcdb13b1f2da126cdfb0aa61a1a9549021 WatchSource:0}: Error finding container 6786561f02672e12e4efcba9926772dcdb13b1f2da126cdfb0aa61a1a9549021: Status 404 returned error can't find the container with id 6786561f02672e12e4efcba9926772dcdb13b1f2da126cdfb0aa61a1a9549021
Nov 03 16:22:13.030530 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.030503    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"]
Nov 03 16:22:13.032251 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:13.032222    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6e4908_9236_4452_a3e1_3f26a5fc0449.slice/crio-38ea8f04bda53551ce91a11b463ed7f82d43eae941f82e6726f56ed3d412c5ca WatchSource:0}: Error finding container 38ea8f04bda53551ce91a11b463ed7f82d43eae941f82e6726f56ed3d412c5ca: Status 404 returned error can't find the container with id 38ea8f04bda53551ce91a11b463ed7f82d43eae941f82e6726f56ed3d412c5ca
Nov 03 16:22:13.172285 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.172249    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp" event={"ID":"9cc4c08f-e98d-49ed-a69e-41fc144d0273","Type":"ContainerStarted","Data":"36e4c8b09da7d71e6b340ae9916eaaf814ad2b02f11d72073d5f93b52ab457c7"}
Nov 03 16:22:13.172715 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.172392    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:13.173934 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.173908    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz" event={"ID":"e5229c4c-b2fb-4418-ab4c-89458192001e","Type":"ContainerStarted","Data":"bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff"}
Nov 03 16:22:13.174055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.173950    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz" event={"ID":"e5229c4c-b2fb-4418-ab4c-89458192001e","Type":"ContainerStarted","Data":"6786561f02672e12e4efcba9926772dcdb13b1f2da126cdfb0aa61a1a9549021"}
Nov 03 16:22:13.174055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.173976    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:13.175406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.175385    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b" event={"ID":"2c6e4908-9236-4452-a3e1-3f26a5fc0449","Type":"ContainerStarted","Data":"da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478"}
Nov 03 16:22:13.175535 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.175433    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b" event={"ID":"2c6e4908-9236-4452-a3e1-3f26a5fc0449","Type":"ContainerStarted","Data":"38ea8f04bda53551ce91a11b463ed7f82d43eae941f82e6726f56ed3d412c5ca"}
Nov 03 16:22:13.175535 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.175477    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:13.286843 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.286744    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp" podStartSLOduration=2.443237075 podStartE2EDuration="3.286729713s" totalImagesPullingTime="843.492638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:13.285991324 +0000 UTC m=+694.865295206" watchObservedRunningTime="2026-11-03 16:22:13.286729713 +0000 UTC m=+694.866033568"
Nov 03 16:22:13.399174 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.399118    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz" podStartSLOduration=1.399103167 podStartE2EDuration="1.399103167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:13.397841198 +0000 UTC m=+694.977145097" watchObservedRunningTime="2026-11-03 16:22:13.399103167 +0000 UTC m=+694.978407073"
Nov 03 16:22:13.512155 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:13.512102    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b" podStartSLOduration=1.51208773 podStartE2EDuration="1.51208773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:13.510601563 +0000 UTC m=+695.089905438" watchObservedRunningTime="2026-11-03 16:22:13.51208773 +0000 UTC m=+695.091391639"
Nov 03 16:22:14.146558 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.146527    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:22:14.804852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.804816    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"]
Nov 03 16:22:14.808904 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.808888    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:14.862145 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.862113    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-192564-dockercfg-htfz6\""
Nov 03 16:22:14.864312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.864282    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"]
Nov 03 16:22:14.885167 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.885128    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866a99-1153-465a-90d6-06a647a8fda6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:14.885329 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.885192    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxnh\" (UniqueName: \"kubernetes.io/projected/c0866a99-1153-465a-90d6-06a647a8fda6-kube-api-access-vbxnh\") pod \"payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:14.907927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.907894    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-192564\""
Nov 03 16:22:14.958207 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.958173    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"]
Nov 03 16:22:14.962244 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.962226    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:14.985856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.985829    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866a99-1153-465a-90d6-06a647a8fda6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:14.986009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.985863    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxnh\" (UniqueName: \"kubernetes.io/projected/c0866a99-1153-465a-90d6-06a647a8fda6-kube-api-access-vbxnh\") pod \"payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:14.986558 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:14.986534    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866a99-1153-465a-90d6-06a647a8fda6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:15.019005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.018978    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"]
Nov 03 16:22:15.042999 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.042958    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxnh\" (UniqueName: \"kubernetes.io/projected/c0866a99-1153-465a-90d6-06a647a8fda6-kube-api-access-vbxnh\") pod \"payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:15.086815 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.086741    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f710392e-71fa-4bcf-af87-96ec3ea904c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:15.086815 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.086786    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxsk\" (UniqueName: \"kubernetes.io/projected/f710392e-71fa-4bcf-af87-96ec3ea904c3-kube-api-access-7sxsk\") pod \"payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:15.122801 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.122768    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:15.189246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.187914    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f710392e-71fa-4bcf-af87-96ec3ea904c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:15.189246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.187982    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxsk\" (UniqueName: \"kubernetes.io/projected/f710392e-71fa-4bcf-af87-96ec3ea904c3-kube-api-access-7sxsk\") pod \"payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:15.190272 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.190239    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f710392e-71fa-4bcf-af87-96ec3ea904c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:15.244483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.244446    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxsk\" (UniqueName: \"kubernetes.io/projected/f710392e-71fa-4bcf-af87-96ec3ea904c3-kube-api-access-7sxsk\") pod \"payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:15.276372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.276341    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:15.497787 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.497760    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"]
Nov 03 16:22:15.500306 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:15.500279    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0866a99_1153_465a_90d6_06a647a8fda6.slice/crio-079223ceca241aad7f9460dbaf1b975094693805c47118752e1239460f1d4480 WatchSource:0}: Error finding container 079223ceca241aad7f9460dbaf1b975094693805c47118752e1239460f1d4480: Status 404 returned error can't find the container with id 079223ceca241aad7f9460dbaf1b975094693805c47118752e1239460f1d4480
Nov 03 16:22:15.597072 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.596999    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"]
Nov 03 16:22:15.600913 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.600893    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.652807 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.652776    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-192564-serving-cert\""
Nov 03 16:22:15.659820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.659788    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"]
Nov 03 16:22:15.665144 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.663511    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"]
Nov 03 16:22:15.665144 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:15.663931    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf710392e_71fa_4bcf_af87_96ec3ea904c3.slice/crio-2564a17ff0274824974b4839e7f70d22343bd6723e72003e59e65694ad230cc6 WatchSource:0}: Error finding container 2564a17ff0274824974b4839e7f70d22343bd6723e72003e59e65694ad230cc6: Status 404 returned error can't find the container with id 2564a17ff0274824974b4839e7f70d22343bd6723e72003e59e65694ad230cc6
Nov 03 16:22:15.694502 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.694468    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6frp\" (UniqueName: \"kubernetes.io/projected/72538d58-0638-40b4-8c19-fe9a18259d2a-kube-api-access-k6frp\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.694659 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.694642    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-metrics-certs\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.694758 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.694723    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72538d58-0638-40b4-8c19-fe9a18259d2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.694888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.694870    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.795985 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.795945    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-metrics-certs\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.796154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.796008    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72538d58-0638-40b4-8c19-fe9a18259d2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.796154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.796092    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.796154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.796127    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6frp\" (UniqueName: \"kubernetes.io/projected/72538d58-0638-40b4-8c19-fe9a18259d2a-kube-api-access-k6frp\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.796383 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:15.796276    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-serving-cert: secret "maas-api-e2e-worker-w3-192564-serving-cert" not found
Nov 03 16:22:15.796383 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:15.796357    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls podName:72538d58-0638-40b4-8c19-fe9a18259d2a nodeName:}" failed. No retries permitted until 2026-11-03 16:22:16.296334493 +0000 UTC m=+697.875638357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls") pod "maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd" (UID: "72538d58-0638-40b4-8c19-fe9a18259d2a") : secret "maas-api-e2e-worker-w3-192564-serving-cert" not found
Nov 03 16:22:15.796675 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.796653    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72538d58-0638-40b4-8c19-fe9a18259d2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.798506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.798487    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-metrics-certs\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:15.853141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:15.853069    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6frp\" (UniqueName: \"kubernetes.io/projected/72538d58-0638-40b4-8c19-fe9a18259d2a-kube-api-access-k6frp\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:16.201873 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.201830    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" event={"ID":"c0866a99-1153-465a-90d6-06a647a8fda6","Type":"ContainerStarted","Data":"d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1"}
Nov 03 16:22:16.202331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.201890    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:16.202331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.201911    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" event={"ID":"c0866a99-1153-465a-90d6-06a647a8fda6","Type":"ContainerStarted","Data":"079223ceca241aad7f9460dbaf1b975094693805c47118752e1239460f1d4480"}
Nov 03 16:22:16.203508 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.203481    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8" event={"ID":"f710392e-71fa-4bcf-af87-96ec3ea904c3","Type":"ContainerStarted","Data":"e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e"}
Nov 03 16:22:16.203604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.203526    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8" event={"ID":"f710392e-71fa-4bcf-af87-96ec3ea904c3","Type":"ContainerStarted","Data":"2564a17ff0274824974b4839e7f70d22343bd6723e72003e59e65694ad230cc6"}
Nov 03 16:22:16.203604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.203547    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:16.301438 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.301403    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:16.304030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.304002    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:16.317893 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.317846    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" podStartSLOduration=2.317828669 podStartE2EDuration="2.317828669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:16.314528713 +0000 UTC m=+697.893832601" watchObservedRunningTime="2026-11-03 16:22:16.317828669 +0000 UTC m=+697.897132525"
Nov 03 16:22:16.429103 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.429046    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8" podStartSLOduration=2.429029892 podStartE2EDuration="2.429029892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:16.428457945 +0000 UTC m=+698.007761816" watchObservedRunningTime="2026-11-03 16:22:16.429029892 +0000 UTC m=+698.008333748"
Nov 03 16:22:16.514454 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.514382    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:16.937028 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:16.936992    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"]
Nov 03 16:22:16.938466 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:16.938436    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72538d58_0638_40b4_8c19_fe9a18259d2a.slice/crio-0d23c9c0b7babb89ff65d5766544fdd06f68401fde6687a15e8bce49f187540a WatchSource:0}: Error finding container 0d23c9c0b7babb89ff65d5766544fdd06f68401fde6687a15e8bce49f187540a: Status 404 returned error can't find the container with id 0d23c9c0b7babb89ff65d5766544fdd06f68401fde6687a15e8bce49f187540a
Nov 03 16:22:17.125715 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:17.125686    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 16:22:17.125855 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:17.125763    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 16:22:17.211979 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:17.211868    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd" event={"ID":"72538d58-0638-40b4-8c19-fe9a18259d2a","Type":"ContainerStarted","Data":"0d23c9c0b7babb89ff65d5766544fdd06f68401fde6687a15e8bce49f187540a"}
Nov 03 16:22:18.219002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:18.218969    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd" event={"ID":"72538d58-0638-40b4-8c19-fe9a18259d2a","Type":"ContainerStarted","Data":"87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9"}
Nov 03 16:22:18.219402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:18.219106    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:18.333472 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:18.333418    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd" podStartSLOduration=2.474001794 podStartE2EDuration="3.333404226s" totalImagesPullingTime="859.402432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:18.332232527 +0000 UTC m=+699.911536376" watchObservedRunningTime="2026-11-03 16:22:18.333404226 +0000 UTC m=+699.912708080"
Nov 03 16:22:19.188549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.188518    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 16:22:19.880605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.880560    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"]
Nov 03 16:22:19.885561 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.885537    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:19.939024 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.938998    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2bb620\""
Nov 03 16:22:19.941047 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.941025    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"]
Nov 03 16:22:19.985123 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.985091    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2bb620-dockercfg-c5mnq\""
Nov 03 16:22:19.989213 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.989178    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"]
Nov 03 16:22:19.993264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:19.993239    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.042142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.042117    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:20.042301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.042156    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvcp5\" (UniqueName: \"kubernetes.io/projected/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-kube-api-access-zvcp5\") pod \"payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:20.046895 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.046863    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"]
Nov 03 16:22:20.142771 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.142688    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pv69\" (UniqueName: \"kubernetes.io/projected/59f795ab-eafb-46bd-969d-551771cdbd7a-kube-api-access-6pv69\") pod \"payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.142771 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.142746    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f795ab-eafb-46bd-969d-551771cdbd7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.142991 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.142921    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:20.142991 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.142963    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvcp5\" (UniqueName: \"kubernetes.io/projected/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-kube-api-access-zvcp5\") pod \"payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:20.143746 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.143723    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:20.200294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.200267    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvcp5\" (UniqueName: \"kubernetes.io/projected/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-kube-api-access-zvcp5\") pod \"payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:20.243660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.243628    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f795ab-eafb-46bd-969d-551771cdbd7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.243802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.243759    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pv69\" (UniqueName: \"kubernetes.io/projected/59f795ab-eafb-46bd-969d-551771cdbd7a-kube-api-access-6pv69\") pod \"payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.244571 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.244547    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f795ab-eafb-46bd-969d-551771cdbd7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.300470 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.300444    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pv69\" (UniqueName: \"kubernetes.io/projected/59f795ab-eafb-46bd-969d-551771cdbd7a-kube-api-access-6pv69\") pod \"payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.307315 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.307288    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:20.500024 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.499994    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:20.689478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.689453    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"]
Nov 03 16:22:20.690131 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:20.690100    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f795ab_eafb_46bd_969d_551771cdbd7a.slice/crio-5354fbb9072f935a448231c3a435e6b785dc0cd87d6c4230afee481e32273da8 WatchSource:0}: Error finding container 5354fbb9072f935a448231c3a435e6b785dc0cd87d6c4230afee481e32273da8: Status 404 returned error can't find the container with id 5354fbb9072f935a448231c3a435e6b785dc0cd87d6c4230afee481e32273da8
Nov 03 16:22:20.870794 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:20.870766    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"]
Nov 03 16:22:20.871649 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:20.871624    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cb2a2f_e7e9_4ec2_90b3_4c39bec86c08.slice/crio-c2e09a13fcdbfb9f004653508edca9b160ee939cbda108cf14dcbaa4e5618884 WatchSource:0}: Error finding container c2e09a13fcdbfb9f004653508edca9b160ee939cbda108cf14dcbaa4e5618884: Status 404 returned error can't find the container with id c2e09a13fcdbfb9f004653508edca9b160ee939cbda108cf14dcbaa4e5618884
Nov 03 16:22:21.238481 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.238449    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" event={"ID":"59f795ab-eafb-46bd-969d-551771cdbd7a","Type":"ContainerStarted","Data":"bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098"}
Nov 03 16:22:21.239154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.238503    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" event={"ID":"59f795ab-eafb-46bd-969d-551771cdbd7a","Type":"ContainerStarted","Data":"5354fbb9072f935a448231c3a435e6b785dc0cd87d6c4230afee481e32273da8"}
Nov 03 16:22:21.239154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.238781    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:21.240412 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.240387    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" event={"ID":"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08","Type":"ContainerStarted","Data":"0d460b439bd147f49b2714f7608e4d6b0789d70a847d197b7580b298fe49a8b0"}
Nov 03 16:22:21.240534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.240435    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" event={"ID":"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08","Type":"ContainerStarted","Data":"c2e09a13fcdbfb9f004653508edca9b160ee939cbda108cf14dcbaa4e5618884"}
Nov 03 16:22:21.240534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.240478    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:21.351021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.350969    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" podStartSLOduration=2.350947212 podStartE2EDuration="2.350947212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:21.350103903 +0000 UTC m=+702.929407758" watchObservedRunningTime="2026-11-03 16:22:21.350947212 +0000 UTC m=+702.930251066"
Nov 03 16:22:21.462081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:21.462020    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" podStartSLOduration=2.462006598 podStartE2EDuration="2.462006598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:21.460814781 +0000 UTC m=+703.040118635" watchObservedRunningTime="2026-11-03 16:22:21.462006598 +0000 UTC m=+703.041310503"
Nov 03 16:22:22.168281 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:22.168250    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:22:22.168424 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:22.168311    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:22:24.184345 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.184316    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 16:22:24.184826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.184810    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 16:22:24.230411 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.230388    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:22:24.929428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.929390    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"]
Nov 03 16:22:24.935602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.935580    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:24.987077 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.987051    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-bcd756\""
Nov 03 16:22:24.987396 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.987372    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-bcd756-dockercfg-7zthm\""
Nov 03 16:22:24.987611 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:24.987590    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"]
Nov 03 16:22:25.090721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.090687    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rplv\" (UniqueName: \"kubernetes.io/projected/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-kube-api-access-8rplv\") pod \"payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:25.090865 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.090739    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:25.192408 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.192334    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:25.192749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.192469    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rplv\" (UniqueName: \"kubernetes.io/projected/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-kube-api-access-8rplv\") pod \"payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:25.193268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.193248    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:25.249397 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.249361    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rplv\" (UniqueName: \"kubernetes.io/projected/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-kube-api-access-8rplv\") pod \"payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:25.251026 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.251002    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:25.627230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:25.627182    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"]
Nov 03 16:22:25.628886 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:25.628856    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2075f8cb_c6c3_4c64_ac44_f153d43aeb70.slice/crio-fdf398a7c545d0df6dbe46a35351f53c6efdd25ec865f323ff8a640911960d79 WatchSource:0}: Error finding container fdf398a7c545d0df6dbe46a35351f53c6efdd25ec865f323ff8a640911960d79: Status 404 returned error can't find the container with id fdf398a7c545d0df6dbe46a35351f53c6efdd25ec865f323ff8a640911960d79
Nov 03 16:22:26.275005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:26.274955    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w" event={"ID":"2075f8cb-c6c3-4c64-ac44-f153d43aeb70","Type":"ContainerStarted","Data":"1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c"}
Nov 03 16:22:26.275435 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:26.275028    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w" event={"ID":"2075f8cb-c6c3-4c64-ac44-f153d43aeb70","Type":"ContainerStarted","Data":"fdf398a7c545d0df6dbe46a35351f53c6efdd25ec865f323ff8a640911960d79"}
Nov 03 16:22:26.275435 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:26.275166    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:26.475891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:26.475792    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w" podStartSLOduration=2.475778042 podStartE2EDuration="2.475778042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:26.474820425 +0000 UTC m=+708.054124303" watchObservedRunningTime="2026-11-03 16:22:26.475778042 +0000 UTC m=+708.055081896"
Nov 03 16:22:27.215312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:27.215280    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:22:27.215491 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:27.215451    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:22:32.249458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:32.249429    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:22:32.249963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:32.249505    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:22:37.285614 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:37.285586    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 16:22:37.884868 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:37.884834    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:22:41.535672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.535638    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"]
Nov 03 16:22:41.541360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.541338    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.639338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.639297    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-547a38-serving-cert\""
Nov 03 16:22:41.642527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.642503    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"]
Nov 03 16:22:41.660151 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.660113    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.660402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.660243    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpc9z\" (UniqueName: \"kubernetes.io/projected/ae03bf56-8bee-4c18-94e6-351b4dedba85-kube-api-access-tpc9z\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.660402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.660394    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-metrics-certs\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.660556 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.660471    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae03bf56-8bee-4c18-94e6-351b4dedba85-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.761770 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.761732    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpc9z\" (UniqueName: \"kubernetes.io/projected/ae03bf56-8bee-4c18-94e6-351b4dedba85-kube-api-access-tpc9z\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.761932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.761796    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-metrics-certs\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.761932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.761828    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae03bf56-8bee-4c18-94e6-351b4dedba85-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.761932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.761878    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.762039 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:41.761999    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-serving-cert: secret "maas-api-e2e-worker-w6-547a38-serving-cert" not found
Nov 03 16:22:41.762072 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:41.762065    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls podName:ae03bf56-8bee-4c18-94e6-351b4dedba85 nodeName:}" failed. No retries permitted until 2026-11-03 16:22:42.262044216 +0000 UTC m=+723.841348068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls") pod "maas-api-e2e-worker-w6-547a38-864568645c-x76tb" (UID: "ae03bf56-8bee-4c18-94e6-351b4dedba85") : secret "maas-api-e2e-worker-w6-547a38-serving-cert" not found
Nov 03 16:22:41.762626 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.762602    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae03bf56-8bee-4c18-94e6-351b4dedba85-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.764261 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.764236    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-metrics-certs\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:41.823135 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:41.823058    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpc9z\" (UniqueName: \"kubernetes.io/projected/ae03bf56-8bee-4c18-94e6-351b4dedba85-kube-api-access-tpc9z\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:42.274590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:42.274556    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:42.276912 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:42.276889    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls\") pod \"maas-api-e2e-worker-w6-547a38-864568645c-x76tb\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:42.455297 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:42.455263    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:42.834581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:42.834553    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"]
Nov 03 16:22:42.835191 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:42.835168    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae03bf56_8bee_4c18_94e6_351b4dedba85.slice/crio-6b659b92eb91ee0c28ffb9ef22c74a0d1b93cc8e1ff2bda45c69520a8f10a388 WatchSource:0}: Error finding container 6b659b92eb91ee0c28ffb9ef22c74a0d1b93cc8e1ff2bda45c69520a8f10a388: Status 404 returned error can't find the container with id 6b659b92eb91ee0c28ffb9ef22c74a0d1b93cc8e1ff2bda45c69520a8f10a388
Nov 03 16:22:43.383907 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:43.383872    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb" event={"ID":"ae03bf56-8bee-4c18-94e6-351b4dedba85","Type":"ContainerStarted","Data":"6b659b92eb91ee0c28ffb9ef22c74a0d1b93cc8e1ff2bda45c69520a8f10a388"}
Nov 03 16:22:44.393780 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:44.393741    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb" event={"ID":"ae03bf56-8bee-4c18-94e6-351b4dedba85","Type":"ContainerStarted","Data":"5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0"}
Nov 03 16:22:44.394179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:44.393839    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:44.512549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:44.512503    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb" podStartSLOduration=2.639848813 podStartE2EDuration="3.512489705s" totalImagesPullingTime="872.640892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:44.511018513 +0000 UTC m=+726.090322368" watchObservedRunningTime="2026-11-03 16:22:44.512489705 +0000 UTC m=+726.091793559"
Nov 03 16:22:50.405230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:50.405150    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:22:51.996013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:51.995979    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"]
Nov 03 16:22:51.999906 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:51.999884    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.052339 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.052316    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-bcd756-serving-cert\""
Nov 03 16:22:52.080421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.080387    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.080549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.080500    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf48z\" (UniqueName: \"kubernetes.io/projected/8052eca5-aedb-4c3d-b37f-051c24042490-kube-api-access-cf48z\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.080614 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.080582    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052eca5-aedb-4c3d-b37f-051c24042490-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.080678 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.080626    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-metrics-certs\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.101277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.101253    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"]
Nov 03 16:22:52.181257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.181231    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052eca5-aedb-4c3d-b37f-051c24042490-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.181412 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.181265    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-metrics-certs\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.181412 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.181323    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.181533 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.181409    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf48z\" (UniqueName: \"kubernetes.io/projected/8052eca5-aedb-4c3d-b37f-051c24042490-kube-api-access-cf48z\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.181533 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:52.181492    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-serving-cert: secret "maas-api-e2e-worker-w1-bcd756-serving-cert" not found
Nov 03 16:22:52.181645 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:22:52.181575    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls podName:8052eca5-aedb-4c3d-b37f-051c24042490 nodeName:}" failed. No retries permitted until 2026-11-03 16:22:52.681554841 +0000 UTC m=+734.260858680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls") pod "maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf" (UID: "8052eca5-aedb-4c3d-b37f-051c24042490") : secret "maas-api-e2e-worker-w1-bcd756-serving-cert" not found
Nov 03 16:22:52.181942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.181917    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052eca5-aedb-4c3d-b37f-051c24042490-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.183731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.183711    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-metrics-certs\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.238455 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.238434    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf48z\" (UniqueName: \"kubernetes.io/projected/8052eca5-aedb-4c3d-b37f-051c24042490-kube-api-access-cf48z\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.685872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.685826    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.688325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.688296    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls\") pod \"maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:52.913360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:52.913327    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:53.289561 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:22:53.289532    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8052eca5_aedb_4c3d_b37f_051c24042490.slice/crio-519ccc2ecedcd1e801ff08cd80f99116848e5f3a028958c5065e80fc46b97eca WatchSource:0}: Error finding container 519ccc2ecedcd1e801ff08cd80f99116848e5f3a028958c5065e80fc46b97eca: Status 404 returned error can't find the container with id 519ccc2ecedcd1e801ff08cd80f99116848e5f3a028958c5065e80fc46b97eca
Nov 03 16:22:53.291379 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:53.291353    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"]
Nov 03 16:22:53.454525 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:53.454491    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf" event={"ID":"8052eca5-aedb-4c3d-b37f-051c24042490","Type":"ContainerStarted","Data":"519ccc2ecedcd1e801ff08cd80f99116848e5f3a028958c5065e80fc46b97eca"}
Nov 03 16:22:54.461332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:54.461243    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf" event={"ID":"8052eca5-aedb-4c3d-b37f-051c24042490","Type":"ContainerStarted","Data":"2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258"}
Nov 03 16:22:54.461698 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:54.461342    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:22:54.575138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:22:54.575089    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf" podStartSLOduration=2.694325609 podStartE2EDuration="3.575077849s" totalImagesPullingTime="880.75224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:22:54.57444121 +0000 UTC m=+736.153745084" watchObservedRunningTime="2026-11-03 16:22:54.575077849 +0000 UTC m=+736.154381745"
Nov 03 16:23:00.472900 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:23:00.472860    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 16:23:07.883689 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:23:07.883651    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:23:37.883951 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:23:37.883915    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:24:07.884300 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:24:07.884248    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:24:19.897994 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.897907    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"]
Nov 03 16:24:19.902406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.902385    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:19.980891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.980865    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de833339-f19a-4c5c-8c68-78aa51789c40-tls-certs\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:19.981035 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.981008    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5g2\" (UniqueName: \"kubernetes.io/projected/de833339-f19a-4c5c-8c68-78aa51789c40-kube-api-access-fg5g2\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:19.981113 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.981039    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-tmp-dir\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:19.981113 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.981104    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-dshm\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:19.981259 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.981148    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-home\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:19.981319 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:19.981270    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-model-cache\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.001929 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.001903    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"kube-root-ca.crt\""
Nov 03 16:24:20.001929 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.001911    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"default-dockercfg-wr497\""
Nov 03 16:24:20.002651 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.002631    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"openshift-service-ca.crt\""
Nov 03 16:24:20.002729 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.002680    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c88d54\"/\"premium-w2-c88d54-kserve-self-signed-certs\""
Nov 03 16:24:20.004648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.004630    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"]
Nov 03 16:24:20.081890 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.081860    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-home\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.081902    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-model-cache\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.081935    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de833339-f19a-4c5c-8c68-78aa51789c40-tls-certs\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.082042    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5g2\" (UniqueName: \"kubernetes.io/projected/de833339-f19a-4c5c-8c68-78aa51789c40-kube-api-access-fg5g2\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.082080    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-tmp-dir\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.082132    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-dshm\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.082249    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-home\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082502 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.082323    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-model-cache\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.082502 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.082459    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-tmp-dir\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.084341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.084319    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-dshm\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.084552 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.084534    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de833339-f19a-4c5c-8c68-78aa51789c40-tls-certs\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.139151 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.139116    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5g2\" (UniqueName: \"kubernetes.io/projected/de833339-f19a-4c5c-8c68-78aa51789c40-kube-api-access-fg5g2\") pod \"premium-w2-c88d54-kserve-56cb7cfbf4-894jl\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") " pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.235987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.235906    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:20.611640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.611614    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"]
Nov 03 16:24:20.613114 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:24:20.613084    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde833339_f19a_4c5c_8c68_78aa51789c40.slice/crio-fc1bf1ec592b3496c80405a841c0364531780201d784d3927f5fbfd55e55254a WatchSource:0}: Error finding container fc1bf1ec592b3496c80405a841c0364531780201d784d3927f5fbfd55e55254a: Status 404 returned error can't find the container with id fc1bf1ec592b3496c80405a841c0364531780201d784d3927f5fbfd55e55254a
Nov 03 16:24:20.997117 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.997075    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" event={"ID":"de833339-f19a-4c5c-8c68-78aa51789c40","Type":"ContainerStarted","Data":"aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb"}
Nov 03 16:24:20.997578 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:20.997127    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" event={"ID":"de833339-f19a-4c5c-8c68-78aa51789c40","Type":"ContainerStarted","Data":"fc1bf1ec592b3496c80405a841c0364531780201d784d3927f5fbfd55e55254a"}
Nov 03 16:24:21.110790 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:21.110738    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podStartSLOduration=2.110723598 podStartE2EDuration="2.110723598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:24:21.110320995 +0000 UTC m=+822.689624869" watchObservedRunningTime="2026-11-03 16:24:21.110723598 +0000 UTC m=+822.690027450"
Nov 03 16:24:21.236740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:21.236699    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:30.236581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:30.236552    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:30.249644 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:30.249616    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:31.066237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:24:31.066183    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:24:37.883308 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:24:37.883269    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:25:07.883685 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:25:07.883643    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:25:37.883936 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:25:37.883888    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:25:38.659583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:38.659555    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:25:38.660941 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:38.660918    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:25:38.670948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:38.670927    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:25:38.672110 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:38.672085    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:25:39.303886 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.303847    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"]
Nov 03 16:25:39.309120 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.309100    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.369240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.367178    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"openshift-service-ca.crt\""
Nov 03 16:25:39.371902 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.371797    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"]
Nov 03 16:25:39.407008 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.406962    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"premium-w6-547a38-kserve-self-signed-certs\""
Nov 03 16:25:39.407008 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.406975    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"default-dockercfg-2nfhm\""
Nov 03 16:25:39.407230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.407006    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-547a38\"/\"kube-root-ca.crt\""
Nov 03 16:25:39.418704 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.418672    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-home\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.418829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.418751    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-model-cache\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.418829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.418787    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-dshm\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.418909 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.418892    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348e1e8e-9cc9-4c31-9097-b766fe181877-tls-certs\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.418943 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.418924    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdr9\" (UniqueName: \"kubernetes.io/projected/348e1e8e-9cc9-4c31-9097-b766fe181877-kube-api-access-jfdr9\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.419116 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.419097    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-tmp-dir\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.520511 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.520477    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348e1e8e-9cc9-4c31-9097-b766fe181877-tls-certs\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.520511 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.520518    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdr9\" (UniqueName: \"kubernetes.io/projected/348e1e8e-9cc9-4c31-9097-b766fe181877-kube-api-access-jfdr9\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.520802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.520702    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-tmp-dir\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.520802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.520748    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-home\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.520802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.520771    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-model-cache\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.520802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.520801    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-dshm\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.521086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.521068    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-tmp-dir\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.521155 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.521136    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-model-cache\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.521229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.521158    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-home\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.523003 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.522979    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-dshm\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.523182 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.523161    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348e1e8e-9cc9-4c31-9097-b766fe181877-tls-certs\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.578693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.578626    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdr9\" (UniqueName: \"kubernetes.io/projected/348e1e8e-9cc9-4c31-9097-b766fe181877-kube-api-access-jfdr9\") pod \"premium-w6-547a38-kserve-789d4bc846-7tzlk\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") " pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:39.621464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:39.621435    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:40.000298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:40.000271    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"]
Nov 03 16:25:40.511135 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:40.511100    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" event={"ID":"348e1e8e-9cc9-4c31-9097-b766fe181877","Type":"ContainerStarted","Data":"6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3"}
Nov 03 16:25:40.511637 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:40.511144    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" event={"ID":"348e1e8e-9cc9-4c31-9097-b766fe181877","Type":"ContainerStarted","Data":"8ce2185a5cf802961860e95742db12329948c86a2073afd46d3f0a3e26c1007f"}
Nov 03 16:25:40.621922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:40.621882    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:40.625836 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:40.625790    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podStartSLOduration=1.6257764799999999 podStartE2EDuration="1.62577648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:25:40.623149721 +0000 UTC m=+902.202453577" watchObservedRunningTime="2026-11-03 16:25:40.62577648 +0000 UTC m=+902.205080334"
Nov 03 16:25:49.621615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:49.621529    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:49.634517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:49.634492    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:50.593480 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:50.593445    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:25:55.768256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.768223    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"]
Nov 03 16:25:55.772461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.772441    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.795692 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.795666    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dld4s\" (UniqueName: \"kubernetes.io/projected/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-kube-api-access-dld4s\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.795810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.795712    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-home\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.795810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.795803    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tls-certs\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.795885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.795868    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-model-cache\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.795918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.795903    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tmp-dir\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.795951 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.795926    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-dshm\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.828087 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.828059    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"]
Nov 03 16:25:55.872590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.872559    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"premium-w1-bcd756-kserve-self-signed-certs\""
Nov 03 16:25:55.872958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.872938    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"kube-root-ca.crt\""
Nov 03 16:25:55.873089 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.872978    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"openshift-service-ca.crt\""
Nov 03 16:25:55.873213 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.873046    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bcd756\"/\"default-dockercfg-q6dvs\""
Nov 03 16:25:55.897137 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897096    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dld4s\" (UniqueName: \"kubernetes.io/projected/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-kube-api-access-dld4s\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897324 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897228    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-home\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897361    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tls-certs\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897442    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-model-cache\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897599 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897544    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tmp-dir\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897616    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-dshm\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897765 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897732    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-home\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897780    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-model-cache\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.897894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.897859    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tmp-dir\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.899744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.899722    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-dshm\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.900049 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.900033    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tls-certs\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:55.954343 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:55.954313    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dld4s\" (UniqueName: \"kubernetes.io/projected/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-kube-api-access-dld4s\") pod \"premium-w1-bcd756-kserve-7756bff6c4-pw69w\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") " pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:56.089469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:56.089380    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:25:56.472149 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:56.472121    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"]
Nov 03 16:25:56.473186 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:25:56.473154    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99609b93_a9c5_4641_a96d_2b7e7aa6baa1.slice/crio-9ccfc5fc1e6c9772e336537904f1011e420d52491cf003009ad598e8c4c24e4b WatchSource:0}: Error finding container 9ccfc5fc1e6c9772e336537904f1011e420d52491cf003009ad598e8c4c24e4b: Status 404 returned error can't find the container with id 9ccfc5fc1e6c9772e336537904f1011e420d52491cf003009ad598e8c4c24e4b
Nov 03 16:25:56.624302 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:56.624252    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" event={"ID":"99609b93-a9c5-4641-a96d-2b7e7aa6baa1","Type":"ContainerStarted","Data":"548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23"}
Nov 03 16:25:56.624302 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:56.624303    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" event={"ID":"99609b93-a9c5-4641-a96d-2b7e7aa6baa1","Type":"ContainerStarted","Data":"9ccfc5fc1e6c9772e336537904f1011e420d52491cf003009ad598e8c4c24e4b"}
Nov 03 16:25:56.746723 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:56.746626    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podStartSLOduration=1.74661076 podStartE2EDuration="1.74661076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:25:56.745705502 +0000 UTC m=+918.325009378" watchObservedRunningTime="2026-11-03 16:25:56.74661076 +0000 UTC m=+918.325914660"
Nov 03 16:25:57.089617 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:25:57.089535    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:26:06.090121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:26:06.090083    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:26:06.104103 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:26:06.104063    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:26:06.712060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:26:06.712028    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:26:07.882787 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:26:07.882751    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:26:37.883938 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:26:37.883891    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:27:07.882716 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:27:07.882679    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:27:19.277478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.277378    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"]
Nov 03 16:27:19.281989 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.281962    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.424219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.412276    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"]
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.427815    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.427871    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.427926    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz69z\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-kube-api-access-dz69z\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.427958    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.428019    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.428070    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.428174    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.428228    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.466354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.428261    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.539900    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.539985    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540084    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540128    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540160    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540242    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540275    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540332    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz69z\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-kube-api-access-dz69z\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540363    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.540901 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.540849    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.558221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.541591    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.558221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.542964    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.558221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.543215    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.558221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.543459    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.577224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.572266    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.577224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.574790    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.613973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.613939    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.639162 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.639127    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz69z\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-kube-api-access-dz69z\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2cq96\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:19.897830 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:19.897788    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:20.330952 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:20.330924    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"]
Nov 03 16:27:20.332213 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:27:20.332167    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f69d9d_4c31_4e8e_acb2_f98fb3817236.slice/crio-a32171883d5cfc63b776c4c9e3ad11ec0c0781e8a644e1970a25c2c38153b7d9 WatchSource:0}: Error finding container a32171883d5cfc63b776c4c9e3ad11ec0c0781e8a644e1970a25c2c38153b7d9: Status 404 returned error can't find the container with id a32171883d5cfc63b776c4c9e3ad11ec0c0781e8a644e1970a25c2c38153b7d9
Nov 03 16:27:20.334821 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:20.334783    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:27:20.334908 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:20.334878    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:27:20.334950 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:20.334918    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:27:21.215005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:21.214966    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" event={"ID":"f8f69d9d-4c31-4e8e-acb2-f98fb3817236","Type":"ContainerStarted","Data":"8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015"}
Nov 03 16:27:21.215240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:21.215042    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" event={"ID":"f8f69d9d-4c31-4e8e-acb2-f98fb3817236","Type":"ContainerStarted","Data":"a32171883d5cfc63b776c4c9e3ad11ec0c0781e8a644e1970a25c2c38153b7d9"}
Nov 03 16:27:21.898899 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:21.898861    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:21.901287 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:21.901253    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2cq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:27:21.901441 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:21.901331    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 16:27:22.898260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:22.898224    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2cq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:27:22.898463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:22.898291    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 16:27:23.898884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:23.898847    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2cq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:27:23.899312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:23.898913    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 16:27:24.898830 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:24.898794    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2cq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:27:24.899046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:24.898853    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 16:27:25.899075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:25.899041    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2cq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:27:25.899492 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:25.899095    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 16:27:26.898420 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:26.898384    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2cq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:27:26.898595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:26.898461    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 16:27:27.898332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:27.898298    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2cq96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 16:27:27.898804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:27.898370    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 16:27:29.023191 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:29.023155    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:29.023728 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:29.023709    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:29.024724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:29.024701    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:27:29.240148 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:29.240071    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podStartSLOduration=10.240050846 podStartE2EDuration="10.240050846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:21.381012207 +0000 UTC m=+1002.960316074" watchObservedRunningTime="2026-11-03 16:27:29.240050846 +0000 UTC m=+1010.819354704"
Nov 03 16:27:34.418188 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.418154    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"]
Nov 03 16:27:34.422883 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.422863    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.515880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.515849    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"]
Nov 03 16:27:34.536295 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536263    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536437 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536315    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536437 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536397    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536469    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536498    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgdq\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-kube-api-access-lhgdq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536522    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536537    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536576    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.536791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.536690    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.637917 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.637879    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.637968    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638021    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638052    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638275 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638079    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgdq\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-kube-api-access-lhgdq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638275 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638114    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638275 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638145    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638425 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638350    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638425 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638398    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638461    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638637    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638844 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638812    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638853    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.638922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.638876    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.640559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.640539    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.640700 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.640682    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.696986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.696919    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.697075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.697059    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgdq\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-kube-api-access-lhgdq\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h8bxw\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:34.737172 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:34.737141    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:35.159768 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.159741    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"]
Nov 03 16:27:35.161162 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:27:35.161114    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f0bcb2_04c2_4831_9a63_066d3da3c93f.slice/crio-c89de53f775f489ebf7000c859c89b9c97e37ee4f8a750bb8ae5909c1ec4e8cf WatchSource:0}: Error finding container c89de53f775f489ebf7000c859c89b9c97e37ee4f8a750bb8ae5909c1ec4e8cf: Status 404 returned error can't find the container with id c89de53f775f489ebf7000c859c89b9c97e37ee4f8a750bb8ae5909c1ec4e8cf
Nov 03 16:27:35.163501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.163458    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:27:35.163591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.163531    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:27:35.163591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.163560    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:27:35.315894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.315854    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" event={"ID":"e3f0bcb2-04c2-4831-9a63-066d3da3c93f","Type":"ContainerStarted","Data":"ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3"}
Nov 03 16:27:35.316083 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.315921    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" event={"ID":"e3f0bcb2-04c2-4831-9a63-066d3da3c93f","Type":"ContainerStarted","Data":"c89de53f775f489ebf7000c859c89b9c97e37ee4f8a750bb8ae5909c1ec4e8cf"}
Nov 03 16:27:35.432092 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.431996    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podStartSLOduration=1.431982655 podStartE2EDuration="1.431982655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:27:35.43049698 +0000 UTC m=+1017.009800836" watchObservedRunningTime="2026-11-03 16:27:35.431982655 +0000 UTC m=+1017.011286510"
Nov 03 16:27:35.737432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:35.737346    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:36.739059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:36.739026    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h8bxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 16:27:36.739456 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:36.739078    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 16:27:37.737439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:37.737407    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h8bxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 16:27:37.737618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:37.737463    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 16:27:37.904470 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:27:37.904435    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:27:38.737998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:38.737968    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h8bxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 16:27:38.738262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:38.738021    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 16:27:39.738339 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:39.738305    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h8bxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 16:27:39.738761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:39.738376    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 16:27:40.737763 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:40.737726    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h8bxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 16:27:40.737947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:40.737783    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 16:27:41.737960 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:41.737925    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h8bxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 16:27:41.738386 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:41.738017    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 16:27:42.737930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:42.737900    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h8bxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 16:27:42.738118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:42.737950    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 16:27:43.742881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:43.742850    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:43.743332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:43.743296    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:27:43.744142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:27:43.744122    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:28:07.884985 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:28:07.884948    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:28:12.140704 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.140665    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"]
Nov 03 16:28:12.145052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.145029    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.201941 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.201908    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"]
Nov 03 16:28:12.245423 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.245391    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"kube-root-ca.crt\""
Nov 03 16:28:12.245652 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.245630    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"openshift-service-ca.crt\""
Nov 03 16:28:12.245697 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.245639    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"default-dockercfg-rwp4d\""
Nov 03 16:28:12.245697 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.245640    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2bb620\"/\"distinct-w4-2bb620-kserve-self-signed-certs\""
Nov 03 16:28:12.288526 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.288490    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-model-cache\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.288526 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.288532    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tls-certs\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.288796 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.288717    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tmp-dir\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.288854 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.288817    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-home\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.288935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.288916    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7jh\" (UniqueName: \"kubernetes.io/projected/cfebe23a-01a1-4b27-ad8d-2497166eef3a-kube-api-access-rk7jh\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.289025 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.289010    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-dshm\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.390590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.390547    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tmp-dir\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.390820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.390639    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-home\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.390820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.390704    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7jh\" (UniqueName: \"kubernetes.io/projected/cfebe23a-01a1-4b27-ad8d-2497166eef3a-kube-api-access-rk7jh\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.390820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.390762    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-dshm\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.390820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.390797    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-model-cache\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.391066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.390821    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tls-certs\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.391066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.390951    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tmp-dir\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.391066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.391032    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-home\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.391243 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.391159    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-model-cache\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.392984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.392960    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-dshm\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.393330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.393312    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tls-certs\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.450047 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.450013    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7jh\" (UniqueName: \"kubernetes.io/projected/cfebe23a-01a1-4b27-ad8d-2497166eef3a-kube-api-access-rk7jh\") pod \"distinct-w4-2bb620-kserve-65576bf8dc-vv8fk\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") " pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.458148 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.458114    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:12.833877 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.833848    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"]
Nov 03 16:28:12.835488 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:28:12.835456    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfebe23a_01a1_4b27_ad8d_2497166eef3a.slice/crio-6faaaa6d8f804e5fbc0a7660701fa8d63ca97e73bf236f9bc167498a0612d656 WatchSource:0}: Error finding container 6faaaa6d8f804e5fbc0a7660701fa8d63ca97e73bf236f9bc167498a0612d656: Status 404 returned error can't find the container with id 6faaaa6d8f804e5fbc0a7660701fa8d63ca97e73bf236f9bc167498a0612d656
Nov 03 16:28:12.932568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.932539    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"]
Nov 03 16:28:12.936649 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.936631    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:12.988609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.988583    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6ae690-openshift-default-dockercfg-6s4h2\""
Nov 03 16:28:12.995921 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:12.995892    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"]
Nov 03 16:28:13.096561 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096477    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.096561 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096538    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.096790 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096662    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265f1668-3643-4128-b4c9-a1ca831689ed-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.096790 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096697    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97rq\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-kube-api-access-k97rq\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.096790 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096725    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265f1668-3643-4128-b4c9-a1ca831689ed-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.096954 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096841    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.096954 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096901    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.097077 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.096966    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.097077 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.097012    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.197806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.197775    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.197827    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.197953    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198012    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265f1668-3643-4128-b4c9-a1ca831689ed-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198037    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97rq\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-kube-api-access-k97rq\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198058    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265f1668-3643-4128-b4c9-a1ca831689ed-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198123    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198181    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198253    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198328    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198692    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.198860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.198736    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.199089 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.199062    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265f1668-3643-4128-b4c9-a1ca831689ed-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.199190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.199127    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.200152 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.200127    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.200612 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.200586    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265f1668-3643-4128-b4c9-a1ca831689ed-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.254938 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.254911    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.255047 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.255033    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97rq\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-kube-api-access-k97rq\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.551504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.551464    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:13.579834 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.579793    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" event={"ID":"cfebe23a-01a1-4b27-ad8d-2497166eef3a","Type":"ContainerStarted","Data":"ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28"}
Nov 03 16:28:13.579968 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.579847    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" event={"ID":"cfebe23a-01a1-4b27-ad8d-2497166eef3a","Type":"ContainerStarted","Data":"6faaaa6d8f804e5fbc0a7660701fa8d63ca97e73bf236f9bc167498a0612d656"}
Nov 03 16:28:13.697583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.697533    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podStartSLOduration=1.6975199189999999 podStartE2EDuration="1.697519919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:13.694888905 +0000 UTC m=+1055.274192772" watchObservedRunningTime="2026-11-03 16:28:13.697519919 +0000 UTC m=+1055.276823773"
Nov 03 16:28:13.985550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.985517    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"]
Nov 03 16:28:13.986560 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:28:13.986527    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265f1668_3643_4128_b4c9_a1ca831689ed.slice/crio-c67229299303cf1d7f3c6264df8aa697da3fb5ccb51e28ef854fba2509863367 WatchSource:0}: Error finding container c67229299303cf1d7f3c6264df8aa697da3fb5ccb51e28ef854fba2509863367: Status 404 returned error can't find the container with id c67229299303cf1d7f3c6264df8aa697da3fb5ccb51e28ef854fba2509863367
Nov 03 16:28:13.988973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.988938    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:28:13.989087 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.989015    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:28:13.989087 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:13.989062    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:28:14.458776 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:14.458746    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:14.591348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:14.591315    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" event={"ID":"265f1668-3643-4128-b4c9-a1ca831689ed","Type":"ContainerStarted","Data":"602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922"}
Nov 03 16:28:14.591553 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:14.591364    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" event={"ID":"265f1668-3643-4128-b4c9-a1ca831689ed","Type":"ContainerStarted","Data":"c67229299303cf1d7f3c6264df8aa697da3fb5ccb51e28ef854fba2509863367"}
Nov 03 16:28:14.848719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:14.848613    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podStartSLOduration=2.848599292 podStartE2EDuration="2.848599292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:14.787561712 +0000 UTC m=+1056.366865586" watchObservedRunningTime="2026-11-03 16:28:14.848599292 +0000 UTC m=+1056.427903146"
Nov 03 16:28:15.552581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:15.552546    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:15.554125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:15.554095    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:28:15.554277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:15.554145    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 16:28:16.552711 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:16.552682    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:28:16.553106 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:16.552735    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 16:28:17.552760 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:17.552723    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:28:17.553154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:17.552776    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 16:28:18.552047 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:18.552018    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:28:18.552247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:18.552062    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 16:28:19.552280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:19.552237    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 16:28:19.552663 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:19.552296    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 16:28:20.774148 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:20.774115    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:20.775350 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:20.775328    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:20.776140 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:20.776117    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:28:22.458637 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:22.458595    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:22.471984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:22.471927    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:22.658852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:22.658817    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:28:37.883156 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:28:37.883117    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:28:39.675820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.675787    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"]
Nov 03 16:28:39.680528 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.680507    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.736470 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.736437    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"]
Nov 03 16:28:39.776727 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.776700    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"openshift-service-ca.crt\""
Nov 03 16:28:39.779027 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.779010    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"kube-root-ca.crt\""
Nov 03 16:28:39.779625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.779604    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"default-dockercfg-vxsb2\""
Nov 03 16:28:39.779684 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.779662    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"distinct-w3-192564-kserve-self-signed-certs\""
Nov 03 16:28:39.815136 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.815110    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-model-cache\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.815316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.815159    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-dshm\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.815368 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.815342    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tmp-dir\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.815475 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.815456    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tls-certs\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.815519 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.815496    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-home\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.815571 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.815555    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrg8\" (UniqueName: \"kubernetes.io/projected/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-kube-api-access-6vrg8\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.916575 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.916542    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tls-certs\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.916575 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.916581    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-home\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.916775 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.916623    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrg8\" (UniqueName: \"kubernetes.io/projected/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-kube-api-access-6vrg8\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.916775 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.916653    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-model-cache\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.916775 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.916677    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-dshm\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.916775 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.916724    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tmp-dir\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.917131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.917098    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-home\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.917296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.917113    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-model-cache\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.917296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.917188    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tmp-dir\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.918938 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.918911    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-dshm\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.919057 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.919041    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tls-certs\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.972828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.972762    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrg8\" (UniqueName: \"kubernetes.io/projected/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-kube-api-access-6vrg8\") pod \"distinct-w3-192564-kserve-6bb594bdb7-66mls\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") " pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:39.993838 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:39.993815    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:40.386761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:40.386735    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"]
Nov 03 16:28:40.387612 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:28:40.387568    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95d8b397_8b41_42c3_8bdf_8ef23ee8663d.slice/crio-83c0d2770a893a0c9a35102c0da51393428d7f34b2814887f6c613a643e335eb WatchSource:0}: Error finding container 83c0d2770a893a0c9a35102c0da51393428d7f34b2814887f6c613a643e335eb: Status 404 returned error can't find the container with id 83c0d2770a893a0c9a35102c0da51393428d7f34b2814887f6c613a643e335eb
Nov 03 16:28:40.764855 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:40.764813    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" event={"ID":"95d8b397-8b41-42c3-8bdf-8ef23ee8663d","Type":"ContainerStarted","Data":"b1fd8192b3fded90e832dda40ed01f09f6dd1b474e3860146656158f2073729d"}
Nov 03 16:28:40.765259 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:40.764863    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" event={"ID":"95d8b397-8b41-42c3-8bdf-8ef23ee8663d","Type":"ContainerStarted","Data":"83c0d2770a893a0c9a35102c0da51393428d7f34b2814887f6c613a643e335eb"}
Nov 03 16:28:40.881537 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:40.881486    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podStartSLOduration=1.881471668 podStartE2EDuration="1.881471668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:40.878161939 +0000 UTC m=+1082.457465793" watchObservedRunningTime="2026-11-03 16:28:40.881471668 +0000 UTC m=+1082.460775523"
Nov 03 16:28:40.994323 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:40.994287    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:41.404716 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.404676    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-c2kgs-lgldm"]
Nov 03 16:28:41.409141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.409116    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.464855 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.464817    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c2kgs-lgldm"]
Nov 03 16:28:41.538309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.538277    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95c64c8a-ee9b-4a79-9087-9042552d9898-archives-path\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.538479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.538344    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm57l\" (UniqueName: \"kubernetes.io/projected/95c64c8a-ee9b-4a79-9087-9042552d9898-kube-api-access-xm57l\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.538479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.538372    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-service-ca-bundle\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.538479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.538409    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-trusted-ca-bundle\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.639675 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.639634    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95c64c8a-ee9b-4a79-9087-9042552d9898-archives-path\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.639869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.639723    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm57l\" (UniqueName: \"kubernetes.io/projected/95c64c8a-ee9b-4a79-9087-9042552d9898-kube-api-access-xm57l\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.639869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.639761    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-service-ca-bundle\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.639869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.639807    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-trusted-ca-bundle\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.640162 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.640134    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95c64c8a-ee9b-4a79-9087-9042552d9898-archives-path\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.640509 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.640485    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-service-ca-bundle\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.640590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.640576    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-trusted-ca-bundle\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.696456 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.696387    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm57l\" (UniqueName: \"kubernetes.io/projected/95c64c8a-ee9b-4a79-9087-9042552d9898-kube-api-access-xm57l\") pod \"periodic-gathering-c2kgs-lgldm\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") " pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:41.722807 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:41.722781    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:28:42.097178 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:42.097150    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c2kgs-lgldm"]
Nov 03 16:28:42.098501 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:28:42.098469    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c64c8a_ee9b_4a79_9087_9042552d9898.slice/crio-7a463c948a99e928b8e54f7ae05770964fcf3abd8330df03f66d385847d6dac9 WatchSource:0}: Error finding container 7a463c948a99e928b8e54f7ae05770964fcf3abd8330df03f66d385847d6dac9: Status 404 returned error can't find the container with id 7a463c948a99e928b8e54f7ae05770964fcf3abd8330df03f66d385847d6dac9
Nov 03 16:28:42.780898 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:42.780834    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c2kgs-lgldm" event={"ID":"95c64c8a-ee9b-4a79-9087-9042552d9898","Type":"ContainerStarted","Data":"e4d595408aaec2542f5e28f9ba00f33f4aab0169366d047d5dc7724df902a6c8"}
Nov 03 16:28:42.781080 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:42.780920    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c2kgs-lgldm" event={"ID":"95c64c8a-ee9b-4a79-9087-9042552d9898","Type":"ContainerStarted","Data":"7a463c948a99e928b8e54f7ae05770964fcf3abd8330df03f66d385847d6dac9"}
Nov 03 16:28:42.895439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:42.895390    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-c2kgs-lgldm" podStartSLOduration=1.895377112 podStartE2EDuration="1.895377112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:42.894021425 +0000 UTC m=+1084.473325297" watchObservedRunningTime="2026-11-03 16:28:42.895377112 +0000 UTC m=+1084.474680964"
Nov 03 16:28:43.580069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.580040    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"]
Nov 03 16:28:43.584085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.584065    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.635945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.635921    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2bb620-openshift-default-dockercfg-m62nq\""
Nov 03 16:28:43.660974 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.660949    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe61a3c-41f2-4688-9886-3cc710efb308-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.660990    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.661030    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.661070    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.661091    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.661156    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe61a3c-41f2-4688-9886-3cc710efb308-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.661174    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7h6m\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-kube-api-access-f7h6m\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.661235    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.661347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.661310    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.685094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.685069    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"]
Nov 03 16:28:43.762072 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762041    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe61a3c-41f2-4688-9886-3cc710efb308-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762072 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762075    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7h6m\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-kube-api-access-f7h6m\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762115    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762192    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762251    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe61a3c-41f2-4688-9886-3cc710efb308-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762286    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762339    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762380    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762408    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762590    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762600    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.762956 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762923    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.763077 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.762983    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.763387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.763364    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe61a3c-41f2-4688-9886-3cc710efb308-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.765000 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.764981    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.765080 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.764980    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe61a3c-41f2-4688-9886-3cc710efb308-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.820825 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.820793    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.820951 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.820870    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7h6m\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-kube-api-access-f7h6m\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:43.899374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:43.899336    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:44.325859 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.325824    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"]
Nov 03 16:28:44.326041 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:28:44.326015    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe61a3c_41f2_4688_9886_3cc710efb308.slice/crio-4f436302311f924acb3d5191b1657000212a5088804e13c8e3049ccb634d86d2 WatchSource:0}: Error finding container 4f436302311f924acb3d5191b1657000212a5088804e13c8e3049ccb634d86d2: Status 404 returned error can't find the container with id 4f436302311f924acb3d5191b1657000212a5088804e13c8e3049ccb634d86d2
Nov 03 16:28:44.328541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.328508    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:28:44.328622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.328569    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:28:44.328622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.328598    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:28:44.795309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.795270    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" event={"ID":"abe61a3c-41f2-4688-9886-3cc710efb308","Type":"ContainerStarted","Data":"cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7"}
Nov 03 16:28:44.795808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.795329    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" event={"ID":"abe61a3c-41f2-4688-9886-3cc710efb308","Type":"ContainerStarted","Data":"4f436302311f924acb3d5191b1657000212a5088804e13c8e3049ccb634d86d2"}
Nov 03 16:28:44.900023 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.899991    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:44.913032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:44.912985    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podStartSLOduration=1.912972054 podStartE2EDuration="1.912972054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:28:44.910984009 +0000 UTC m=+1086.490287864" watchObservedRunningTime="2026-11-03 16:28:44.912972054 +0000 UTC m=+1086.492275984"
Nov 03 16:28:45.902029 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:45.901996    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:28:45.902528 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:45.902048    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:28:46.900469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:46.900437    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:28:46.900645 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:46.900496    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:28:47.899835 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:47.899756    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:28:47.899835 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:47.899818    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:28:48.900160 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:48.900120    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:28:48.900582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:48.900191    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:28:49.901641 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:49.901599    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:28:49.902147 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:49.901680    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:28:49.994542 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:49.994507    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:50.007950 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:50.007922    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:50.854901 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:50.854873    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:28:50.900680 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:50.900628    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 16:28:50.900885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:50.900731    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 16:28:51.905861 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:51.905828    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:51.906563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:51.906542    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:28:51.907700 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:28:51.907676    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:29:07.884364 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:29:07.884314    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:29:14.141253 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.141217    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"]
Nov 03 16:29:14.147372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.147346    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.210326 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.210287    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-192564-openshift-default-dockercfg-256th\""
Nov 03 16:29:14.252219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.252170    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.252534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.252508    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.252739 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.252721    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8887665-7239-480d-b6d1-c0352e1edce5-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.253261 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.252876    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.257282 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.253484    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9z5\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-kube-api-access-6p9z5\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.257521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.257501    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8887665-7239-480d-b6d1-c0352e1edce5-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.257662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.257641    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.257813 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.257530    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"]
Nov 03 16:29:14.257813 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.257755    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.257946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.257843    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359397    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359435    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359495    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359514    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359562    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8887665-7239-480d-b6d1-c0352e1edce5-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359597    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359627    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9z5\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-kube-api-access-6p9z5\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.359687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359674    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8887665-7239-480d-b6d1-c0352e1edce5-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.360015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359710    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.360015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.359897    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.360235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.360194    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.360316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.360265    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.360468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.360446    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.360573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.360552    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8887665-7239-480d-b6d1-c0352e1edce5-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.362154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.362125    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.362588 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.362566    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8887665-7239-480d-b6d1-c0352e1edce5-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.449624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.449584    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9z5\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-kube-api-access-6p9z5\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.451740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.451712    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.463208 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.463178    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:14.925575 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.916656    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"]
Nov 03 16:29:14.938965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.936273    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:29:14.938965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.936390    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:29:14.938965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:14.936430    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:29:15.059909 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:15.059860    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" event={"ID":"d8887665-7239-480d-b6d1-c0352e1edce5","Type":"ContainerStarted","Data":"b2572b0fb27a67e412dce04fc102d8e2eff54289e79d950b9a1c94b0f2137cbe"}
Nov 03 16:29:16.094419 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:16.094370    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" event={"ID":"d8887665-7239-480d-b6d1-c0352e1edce5","Type":"ContainerStarted","Data":"c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7"}
Nov 03 16:29:16.227429 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:16.227374    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podStartSLOduration=2.227360395 podStartE2EDuration="2.227360395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:29:16.217964684 +0000 UTC m=+1117.797268551" watchObservedRunningTime="2026-11-03 16:29:16.227360395 +0000 UTC m=+1117.806664241"
Nov 03 16:29:16.466775 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:16.464683    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:16.467092 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:16.466717    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:29:16.467320 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:16.467293    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:29:17.474224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:17.465548    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:29:17.474224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:17.465618    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:29:18.474232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:18.466585    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:29:18.474232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:18.466648    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:29:19.465522 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:19.465485    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:29:19.465814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:19.465786    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:29:20.470220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:20.464498    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:29:20.470220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:20.464555    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:29:21.464743 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:21.464701    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 16:29:21.464955 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:21.464751    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 16:29:22.469483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:22.469450    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:22.469914 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:22.469824    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:22.470567 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:22.470544    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:29:37.886345 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:29:37.886307    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:29:43.407834 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.407797    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"]
Nov 03 16:29:43.413016 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.412993    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.465051 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.465013    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42da0f-openshift-default-dockercfg-l8c4p\""
Nov 03 16:29:43.514852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.514821    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"]
Nov 03 16:29:43.544057 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544025    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544076    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544183    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544322    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fbeb18-4414-43b8-a190-36beb56e4193-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544443    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544511 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544500    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544586    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544660    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.544718 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.544699    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whj5h\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-kube-api-access-whj5h\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645494    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645694 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645532    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645694 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645556    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whj5h\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-kube-api-access-whj5h\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645694 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645637    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645694 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645656    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645694 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645684    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645754    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fbeb18-4414-43b8-a190-36beb56e4193-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645851    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645909    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.645987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.645930    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.646346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.646238    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.646346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.646302    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.646462 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.646388    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.646732 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.646700    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fbeb18-4414-43b8-a190-36beb56e4193-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.648342 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.648315    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.648466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.648343    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.705609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.705543    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.705786 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.705766    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whj5h\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-kube-api-access-whj5h\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:43.729422 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:43.729399    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:44.159826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.159798    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"]
Nov 03 16:29:44.161289 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:29:44.161257    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fbeb18_4414_43b8_a190_36beb56e4193.slice/crio-3b238b5fa2def0d04e69a3abeec5bd5de66c401f781526eaf9e106c347951a5f WatchSource:0}: Error finding container 3b238b5fa2def0d04e69a3abeec5bd5de66c401f781526eaf9e106c347951a5f: Status 404 returned error can't find the container with id 3b238b5fa2def0d04e69a3abeec5bd5de66c401f781526eaf9e106c347951a5f
Nov 03 16:29:44.163675 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.163637    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:29:44.163778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.163704    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:29:44.163778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.163734    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:29:44.283855 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.283813    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" event={"ID":"c3fbeb18-4414-43b8-a190-36beb56e4193","Type":"ContainerStarted","Data":"2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729"}
Nov 03 16:29:44.284054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.283878    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" event={"ID":"c3fbeb18-4414-43b8-a190-36beb56e4193","Type":"ContainerStarted","Data":"3b238b5fa2def0d04e69a3abeec5bd5de66c401f781526eaf9e106c347951a5f"}
Nov 03 16:29:44.399552 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.399500    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podStartSLOduration=1.399480853 podStartE2EDuration="1.399480853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:29:44.398728513 +0000 UTC m=+1145.978032374" watchObservedRunningTime="2026-11-03 16:29:44.399480853 +0000 UTC m=+1145.978784708"
Nov 03 16:29:44.730074 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:44.730041    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:45.732282 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:45.732248    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:29:45.732758 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:45.732305    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:29:46.729745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:46.729707    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:29:46.729935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:46.729757    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:29:47.730481 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:47.730443    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:29:47.730884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:47.730508    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:29:48.730503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:48.730466    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:29:48.730894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:48.730527    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:29:49.730794 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:49.730761    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:29:49.731179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:49.730820    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:29:50.730452 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:50.730405    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 16:29:50.730660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:50.730475    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 16:29:51.734829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:51.734785    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:51.735323 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:51.735240    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:29:51.736086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:29:51.736065    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 16:30:00.213612 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.213575    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"]
Nov 03 16:30:00.217861 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.217839    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.275632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.275597    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4991d3f3-269a-4914-9b1f-b60d107d4b45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.275786 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.275768    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghsvc\" (UniqueName: \"kubernetes.io/projected/4991d3f3-269a-4914-9b1f-b60d107d4b45-kube-api-access-ghsvc\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.339123 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.339086    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"]
Nov 03 16:30:00.377088 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.377050    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghsvc\" (UniqueName: \"kubernetes.io/projected/4991d3f3-269a-4914-9b1f-b60d107d4b45-kube-api-access-ghsvc\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.377308 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.377134    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4991d3f3-269a-4914-9b1f-b60d107d4b45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.377977 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.377953    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4991d3f3-269a-4914-9b1f-b60d107d4b45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.381790 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.381768    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 16:30:00.446992 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.446952    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghsvc\" (UniqueName: \"kubernetes.io/projected/4991d3f3-269a-4914-9b1f-b60d107d4b45-kube-api-access-ghsvc\") pod \"maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.533272 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.533189    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:00.917296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.917116    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"]
Nov 03 16:30:00.919381 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:30:00.919351    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4991d3f3_269a_4914_9b1f_b60d107d4b45.slice/crio-62ad08a5f958b58f29e16c5fb2234aa8d2759bf2723c5924e37e311fb4da418b WatchSource:0}: Error finding container 62ad08a5f958b58f29e16c5fb2234aa8d2759bf2723c5924e37e311fb4da418b: Status 404 returned error can't find the container with id 62ad08a5f958b58f29e16c5fb2234aa8d2759bf2723c5924e37e311fb4da418b
Nov 03 16:30:00.921357 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:00.921333    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:30:01.414544 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:01.414515    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" event={"ID":"4991d3f3-269a-4914-9b1f-b60d107d4b45","Type":"ContainerStarted","Data":"62ad08a5f958b58f29e16c5fb2234aa8d2759bf2723c5924e37e311fb4da418b"}
Nov 03 16:30:07.883653 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:30:07.883564    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:30:08.467828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:08.467743    5112 generic.go:356] "Generic (PLEG): container finished" podID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerID="308b4ba4b2190c811d84bcb802d0bed4087c9db10c864110444dad3b74317dee" exitCode=60
Nov 03 16:30:08.467828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:08.467786    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" event={"ID":"4991d3f3-269a-4914-9b1f-b60d107d4b45","Type":"ContainerDied","Data":"308b4ba4b2190c811d84bcb802d0bed4087c9db10c864110444dad3b74317dee"}
Nov 03 16:30:08.468067 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:08.468050    5112 scope.go:122] "RemoveContainer" containerID="308b4ba4b2190c811d84bcb802d0bed4087c9db10c864110444dad3b74317dee"
Nov 03 16:30:09.476211 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:09.476169    5112 generic.go:356] "Generic (PLEG): container finished" podID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerID="544d69abf8bd663197354a743d293f7cc76f284b99f5cc9c7f9c1d71276f9f35" exitCode=60
Nov 03 16:30:09.476589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:09.476252    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" event={"ID":"4991d3f3-269a-4914-9b1f-b60d107d4b45","Type":"ContainerDied","Data":"544d69abf8bd663197354a743d293f7cc76f284b99f5cc9c7f9c1d71276f9f35"}
Nov 03 16:30:09.476589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:09.476307    5112 scope.go:122] "RemoveContainer" containerID="308b4ba4b2190c811d84bcb802d0bed4087c9db10c864110444dad3b74317dee"
Nov 03 16:30:09.476589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:09.476539    5112 scope.go:122] "RemoveContainer" containerID="544d69abf8bd663197354a743d293f7cc76f284b99f5cc9c7f9c1d71276f9f35"
Nov 03 16:30:09.476859 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:30:09.476836    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk_odh-ai-gateway-infra(4991d3f3-269a-4914-9b1f-b60d107d4b45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45"
Nov 03 16:30:10.486750 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:10.486717    5112 scope.go:122] "RemoveContainer" containerID="544d69abf8bd663197354a743d293f7cc76f284b99f5cc9c7f9c1d71276f9f35"
Nov 03 16:30:10.487229 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:30:10.487083    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk_odh-ai-gateway-infra(4991d3f3-269a-4914-9b1f-b60d107d4b45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45"
Nov 03 16:30:19.549856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:19.549823    5112 generic.go:356] "Generic (PLEG): container finished" podID="95c64c8a-ee9b-4a79-9087-9042552d9898" containerID="e4d595408aaec2542f5e28f9ba00f33f4aab0169366d047d5dc7724df902a6c8" exitCode=0
Nov 03 16:30:19.550310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:19.549891    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c2kgs-lgldm" event={"ID":"95c64c8a-ee9b-4a79-9087-9042552d9898","Type":"ContainerDied","Data":"e4d595408aaec2542f5e28f9ba00f33f4aab0169366d047d5dc7724df902a6c8"}
Nov 03 16:30:20.723120 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.723094    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:30:20.833969 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.833885    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95c64c8a-ee9b-4a79-9087-9042552d9898-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95c64c8a-ee9b-4a79-9087-9042552d9898-archives-path\") pod \"95c64c8a-ee9b-4a79-9087-9042552d9898\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") "
Nov 03 16:30:20.833969 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.833926    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c64c8a-ee9b-4a79-9087-9042552d9898-kube-api-access-xm57l\" (UniqueName: \"kubernetes.io/projected/95c64c8a-ee9b-4a79-9087-9042552d9898-kube-api-access-xm57l\") pod \"95c64c8a-ee9b-4a79-9087-9042552d9898\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") "
Nov 03 16:30:20.834251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.833994    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-trusted-ca-bundle\") pod \"95c64c8a-ee9b-4a79-9087-9042552d9898\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") "
Nov 03 16:30:20.834251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.834166    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-service-ca-bundle\") pod \"95c64c8a-ee9b-4a79-9087-9042552d9898\" (UID: \"95c64c8a-ee9b-4a79-9087-9042552d9898\") "
Nov 03 16:30:20.834383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.834251    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c64c8a-ee9b-4a79-9087-9042552d9898-archives-path" pod "95c64c8a-ee9b-4a79-9087-9042552d9898" (UID: "95c64c8a-ee9b-4a79-9087-9042552d9898"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:30:20.834646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.834622    5112 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95c64c8a-ee9b-4a79-9087-9042552d9898-archives-path\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:20.834839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.834697    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-trusted-ca-bundle" pod "95c64c8a-ee9b-4a79-9087-9042552d9898" (UID: "95c64c8a-ee9b-4a79-9087-9042552d9898"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:30:20.834839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.834728    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-service-ca-bundle" pod "95c64c8a-ee9b-4a79-9087-9042552d9898" (UID: "95c64c8a-ee9b-4a79-9087-9042552d9898"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:30:20.836963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.836937    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c64c8a-ee9b-4a79-9087-9042552d9898-kube-api-access-xm57l" pod "95c64c8a-ee9b-4a79-9087-9042552d9898" (UID: "95c64c8a-ee9b-4a79-9087-9042552d9898"). InnerVolumeSpecName "kube-api-access-xm57l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:30:20.935648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.935588    5112 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-service-ca-bundle\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:20.935648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.935640    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm57l\" (UniqueName: \"kubernetes.io/projected/95c64c8a-ee9b-4a79-9087-9042552d9898-kube-api-access-xm57l\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:20.935648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:20.935656    5112 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95c64c8a-ee9b-4a79-9087-9042552d9898-trusted-ca-bundle\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:21.567466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:21.567437    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c2kgs-lgldm"
Nov 03 16:30:21.567676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:21.567480    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c2kgs-lgldm" event={"ID":"95c64c8a-ee9b-4a79-9087-9042552d9898","Type":"ContainerDied","Data":"7a463c948a99e928b8e54f7ae05770964fcf3abd8330df03f66d385847d6dac9"}
Nov 03 16:30:21.567676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:21.567538    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a463c948a99e928b8e54f7ae05770964fcf3abd8330df03f66d385847d6dac9"
Nov 03 16:30:22.554461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:22.554433    5112 scope.go:122] "RemoveContainer" containerID="544d69abf8bd663197354a743d293f7cc76f284b99f5cc9c7f9c1d71276f9f35"
Nov 03 16:30:23.583626 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:23.583591    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" event={"ID":"4991d3f3-269a-4914-9b1f-b60d107d4b45","Type":"ContainerStarted","Data":"f48373a4d52737492b9129f5ad2703e1f19e90035293fe612a904a4fa7be5efd"}
Nov 03 16:30:23.695804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:23.695752    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" podStartSLOduration=17.331709882 podStartE2EDuration="23.695737874s" totalImagesPullingTime="6.364027992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:23.695224655 +0000 UTC m=+1185.274528501" watchObservedRunningTime="2026-11-03 16:30:23.695737874 +0000 UTC m=+1185.275041729"
Nov 03 16:30:24.592842 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:24.592807    5112 generic.go:356] "Generic (PLEG): container finished" podID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerID="f48373a4d52737492b9129f5ad2703e1f19e90035293fe612a904a4fa7be5efd" exitCode=60
Nov 03 16:30:24.593352 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:24.592875    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" event={"ID":"4991d3f3-269a-4914-9b1f-b60d107d4b45","Type":"ContainerDied","Data":"f48373a4d52737492b9129f5ad2703e1f19e90035293fe612a904a4fa7be5efd"}
Nov 03 16:30:24.593352 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:24.592951    5112 scope.go:122] "RemoveContainer" containerID="544d69abf8bd663197354a743d293f7cc76f284b99f5cc9c7f9c1d71276f9f35"
Nov 03 16:30:24.593352 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:24.593290    5112 scope.go:122] "RemoveContainer" containerID="f48373a4d52737492b9129f5ad2703e1f19e90035293fe612a904a4fa7be5efd"
Nov 03 16:30:24.593568 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:30:24.593549    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk_odh-ai-gateway-infra(4991d3f3-269a-4914-9b1f-b60d107d4b45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45"
Nov 03 16:30:24.717344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:24.717313    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"]
Nov 03 16:30:25.768942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:25.768910    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:25.795431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:25.795398    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4991d3f3-269a-4914-9b1f-b60d107d4b45-kube-api-access-ghsvc\" (UniqueName: \"kubernetes.io/projected/4991d3f3-269a-4914-9b1f-b60d107d4b45-kube-api-access-ghsvc\") pod \"4991d3f3-269a-4914-9b1f-b60d107d4b45\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") "
Nov 03 16:30:25.795590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:25.795569    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4991d3f3-269a-4914-9b1f-b60d107d4b45-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4991d3f3-269a-4914-9b1f-b60d107d4b45-maas-api-service-ca\") pod \"4991d3f3-269a-4914-9b1f-b60d107d4b45\" (UID: \"4991d3f3-269a-4914-9b1f-b60d107d4b45\") "
Nov 03 16:30:25.795973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:25.795943    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4991d3f3-269a-4914-9b1f-b60d107d4b45-maas-api-service-ca" pod "4991d3f3-269a-4914-9b1f-b60d107d4b45" (UID: "4991d3f3-269a-4914-9b1f-b60d107d4b45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:30:25.796139 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:25.796121    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4991d3f3-269a-4914-9b1f-b60d107d4b45-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:25.797839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:25.797807    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4991d3f3-269a-4914-9b1f-b60d107d4b45-kube-api-access-ghsvc" pod "4991d3f3-269a-4914-9b1f-b60d107d4b45" (UID: "4991d3f3-269a-4914-9b1f-b60d107d4b45"). InnerVolumeSpecName "kube-api-access-ghsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:30:25.897658 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:25.897621    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghsvc\" (UniqueName: \"kubernetes.io/projected/4991d3f3-269a-4914-9b1f-b60d107d4b45-kube-api-access-ghsvc\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:30:26.609017 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:26.608985    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"
Nov 03 16:30:26.609222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:26.609017    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk" event={"ID":"4991d3f3-269a-4914-9b1f-b60d107d4b45","Type":"ContainerDied","Data":"62ad08a5f958b58f29e16c5fb2234aa8d2759bf2723c5924e37e311fb4da418b"}
Nov 03 16:30:26.609222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:26.609082    5112 scope.go:122] "RemoveContainer" containerID="f48373a4d52737492b9129f5ad2703e1f19e90035293fe612a904a4fa7be5efd"
Nov 03 16:30:26.769959 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:26.769918    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"]
Nov 03 16:30:26.816898 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:26.816813    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42da0f-29895390-hkmlk"]
Nov 03 16:30:28.550931 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:28.550896    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" path="/var/lib/kubelet/pods/4991d3f3-269a-4914-9b1f-b60d107d4b45/volumes"
Nov 03 16:30:37.883586 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:30:37.883544    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:30:38.708677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:38.708639    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:30:38.711128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:38.711101    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:30:38.720870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:38.720846    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:30:38.723578 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:30:38.723555    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:31:00.564957 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.564919    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"]
Nov 03 16:31:00.565909 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.565886    5112 cpu_manager.go:414] "containerMap: removing container" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:00.565955 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.565915    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:00.565955 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.565934    5112 cpu_manager.go:414] "containerMap: removing container" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:00.565955 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.565942    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:00.566050 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.566010    5112 cpu_manager.go:414] "containerMap: removing container" podUID="95c64c8a-ee9b-4a79-9087-9042552d9898" containerName="insights-gathering"
Nov 03 16:31:00.566050 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.566017    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c64c8a-ee9b-4a79-9087-9042552d9898" containerName="insights-gathering"
Nov 03 16:31:00.566223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.566189    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:00.566278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.566229    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:00.566278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.566243    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:00.566278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.566264    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c64c8a-ee9b-4a79-9087-9042552d9898" containerName="insights-gathering"
Nov 03 16:31:00.570534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.570511    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.632487 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.632453    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-model-cache\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.632662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.632560    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-tmp-dir\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.632662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.632623    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-dshm\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.632662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.632642    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474dd53-f607-4a36-880a-670a21b9a1d8-tls-certs\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.632818 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.632738    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-home\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.632965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.632940    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk7z\" (UniqueName: \"kubernetes.io/projected/1474dd53-f607-4a36-880a-670a21b9a1d8-kube-api-access-6rk7z\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.668997 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.668952    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-192564\"/\"distinct-2-w3-192564-kserve-self-signed-certs\""
Nov 03 16:31:00.671478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.671446    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"]
Nov 03 16:31:00.734183 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734149    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-model-cache\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.734401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734227    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-tmp-dir\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.734401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734292    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-dshm\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.734401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734315    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474dd53-f607-4a36-880a-670a21b9a1d8-tls-certs\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.734401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734363    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-home\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.734615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734429    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk7z\" (UniqueName: \"kubernetes.io/projected/1474dd53-f607-4a36-880a-670a21b9a1d8-kube-api-access-6rk7z\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.734668 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734642    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-model-cache\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.734958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.734931    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-home\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.735114 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.735076    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-tmp-dir\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.736894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.736863    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-dshm\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.738021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.737997    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474dd53-f607-4a36-880a-670a21b9a1d8-tls-certs\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.791260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.791228    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk7z\" (UniqueName: \"kubernetes.io/projected/1474dd53-f607-4a36-880a-670a21b9a1d8-kube-api-access-6rk7z\") pod \"distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") " pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:00.884903 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:00.884821    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:01.284164 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:01.284126    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"]
Nov 03 16:31:01.284615 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:31:01.284574    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1474dd53_f607_4a36_880a_670a21b9a1d8.slice/crio-c0f1f623cc6db1ea202b292dfd30c9115fe8bb9bfabec45fefe13d49240df085 WatchSource:0}: Error finding container c0f1f623cc6db1ea202b292dfd30c9115fe8bb9bfabec45fefe13d49240df085: Status 404 returned error can't find the container with id c0f1f623cc6db1ea202b292dfd30c9115fe8bb9bfabec45fefe13d49240df085
Nov 03 16:31:01.855256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:01.855179    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" event={"ID":"1474dd53-f607-4a36-880a-670a21b9a1d8","Type":"ContainerStarted","Data":"72a38ab8f40f1176cbce942e5c1be6059e64ae9fa6cca2b1b85fe24ff5279869"}
Nov 03 16:31:01.855766 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:01.855268    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" event={"ID":"1474dd53-f607-4a36-880a-670a21b9a1d8","Type":"ContainerStarted","Data":"c0f1f623cc6db1ea202b292dfd30c9115fe8bb9bfabec45fefe13d49240df085"}
Nov 03 16:31:01.885317 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:01.885278    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:01.970565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:01.970505    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podStartSLOduration=1.9704872820000001 podStartE2EDuration="1.970487282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:01.968504166 +0000 UTC m=+1223.547808022" watchObservedRunningTime="2026-11-03 16:31:01.970487282 +0000 UTC m=+1223.549791137"
Nov 03 16:31:07.885051 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:31:07.885004    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:31:10.885343 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:10.885313    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:10.899004 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:10.898979    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:10.929020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:10.928994    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:31:12.975676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:12.975638    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"]
Nov 03 16:31:12.976672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:12.976645    5112 cpu_manager.go:414] "containerMap: removing container" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:12.976672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:12.976672    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4991d3f3-269a-4914-9b1f-b60d107d4b45" containerName="cleanup"
Nov 03 16:31:12.980592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:12.980573    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.079461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.079391    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"]
Nov 03 16:31:13.079755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.079735    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"openshift-service-ca.crt\""
Nov 03 16:31:13.079805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.079775    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"kube-root-ca.crt\""
Nov 03 16:31:13.080035 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.080016    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"distinct-2-w0-42da0f-kserve-self-signed-certs\""
Nov 03 16:31:13.080086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.080025    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42da0f\"/\"default-dockercfg-jqwqn\""
Nov 03 16:31:13.091940 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.091911    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-home\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.092092 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.092045    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tls-certs\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.092157 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.092135    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tmp-dir\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.092285 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.092261    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-model-cache\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.092381 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.092358    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-dshm\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.092433 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.092404    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnd5\" (UniqueName: \"kubernetes.io/projected/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-kube-api-access-rjnd5\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.193661 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.193618    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-model-cache\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.193851 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.193708    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-dshm\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.193851 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.193751    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnd5\" (UniqueName: \"kubernetes.io/projected/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-kube-api-access-rjnd5\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.194010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.193869    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-home\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.194010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.193974    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tls-certs\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.194010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.194003    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tmp-dir\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.194165 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.194092    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-model-cache\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.194800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.194416    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tmp-dir\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.194800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.194681    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-home\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.196605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.196578    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-dshm\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.196734 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.196588    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tls-certs\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.254603 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.254521    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnd5\" (UniqueName: \"kubernetes.io/projected/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-kube-api-access-rjnd5\") pod \"distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") " pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.295373 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.295332    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:13.716302 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.716271    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"]
Nov 03 16:31:13.718070 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:31:13.718032    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7af5ace_a5cd_4b3b_b88b_61969a16ea82.slice/crio-4a902c3eabd28c859ecd7eaead81dcc8da56efcb2ff1a7317bfa6ae430f89143 WatchSource:0}: Error finding container 4a902c3eabd28c859ecd7eaead81dcc8da56efcb2ff1a7317bfa6ae430f89143: Status 404 returned error can't find the container with id 4a902c3eabd28c859ecd7eaead81dcc8da56efcb2ff1a7317bfa6ae430f89143
Nov 03 16:31:13.940890 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.940852    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" event={"ID":"a7af5ace-a5cd-4b3b-b88b-61969a16ea82","Type":"ContainerStarted","Data":"551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34"}
Nov 03 16:31:13.940890 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:13.940897    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" event={"ID":"a7af5ace-a5cd-4b3b-b88b-61969a16ea82","Type":"ContainerStarted","Data":"4a902c3eabd28c859ecd7eaead81dcc8da56efcb2ff1a7317bfa6ae430f89143"}
Nov 03 16:31:14.061271 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:14.061073    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podStartSLOduration=2.061050163 podStartE2EDuration="2.061050163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:14.056344566 +0000 UTC m=+1235.635648425" watchObservedRunningTime="2026-11-03 16:31:14.061050163 +0000 UTC m=+1235.640354020"
Nov 03 16:31:14.296469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:14.296429    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:23.295529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:23.295487    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:23.308564 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:23.308535    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:24.017689 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:24.017658    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:31:30.563468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.563435    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"]
Nov 03 16:31:30.568035 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.568004    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.623149 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.623123    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c88d54-openshift-default-dockercfg-7w7bk\""
Nov 03 16:31:30.684222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.684088    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"]
Nov 03 16:31:30.730010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.729974    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec52cc47-42d7-4e8f-b671-154086f71067-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730015    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730096    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhcz\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-kube-api-access-bxhcz\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730168    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730186    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec52cc47-42d7-4e8f-b671-154086f71067-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730265    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730300    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730409 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730374    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.730457 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.730440    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831144    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831229    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831301    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec52cc47-42d7-4e8f-b671-154086f71067-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831414    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831467    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhcz\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-kube-api-access-bxhcz\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831533    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831549    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec52cc47-42d7-4e8f-b671-154086f71067-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831560    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831578    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831645    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831656    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.831926 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.831848    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.832177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.832156    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.832303 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.832286    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec52cc47-42d7-4e8f-b671-154086f71067-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.833657 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.833634    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.833868 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.833849    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec52cc47-42d7-4e8f-b671-154086f71067-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.875713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.873273    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"]
Nov 03 16:31:30.877934 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.877916    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:30.888914 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.888889    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhcz\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-kube-api-access-bxhcz\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.895037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.895010    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:30.937046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:30.937010    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"]
Nov 03 16:31:31.019765 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.019734    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bcd756-openshift-default-dockercfg-8drrh\""
Nov 03 16:31:31.036429 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036405    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036454    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036603    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036682 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036668    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036720    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036794    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7r28\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-kube-api-access-j7r28\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036828    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036883 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036875    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.036938 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.036927    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16acfc1-e819-4a7c-b026-c7ffde68da12-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138142    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138188    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7r28\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-kube-api-access-j7r28\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138329    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138380    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138557 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138462    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16acfc1-e819-4a7c-b026-c7ffde68da12-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138557 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138504    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138667 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138561    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138669    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138803 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138782    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138897 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138869    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.138990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.138969    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.139115 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.139098    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.139351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.139330    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16acfc1-e819-4a7c-b026-c7ffde68da12-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.139408 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.139390    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.141949 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.141929    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.142177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.142157    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.183084 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.183059    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:31.194356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.194337    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7r28\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-kube-api-access-j7r28\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.194550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.194529    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.494637 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.494599    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:31.611737 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.611705    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"]
Nov 03 16:31:31.642478 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:31:31.642450    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec52cc47_42d7_4e8f_b671_154086f71067.slice/crio-561681769af1b865a93847be0ce59d7c1fdce2a7da8a74f37f08f87d2845422d WatchSource:0}: Error finding container 561681769af1b865a93847be0ce59d7c1fdce2a7da8a74f37f08f87d2845422d: Status 404 returned error can't find the container with id 561681769af1b865a93847be0ce59d7c1fdce2a7da8a74f37f08f87d2845422d
Nov 03 16:31:31.645083 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.645052    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:31:31.645176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.645117    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:31:31.645176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.645157    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:31:31.881176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.881146    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"]
Nov 03 16:31:31.883470 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:31:31.883436    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16acfc1_e819_4a7c_b026_c7ffde68da12.slice/crio-c480b9ea29f681f380919addbebb6abe5961412f2f0bc73b64b4f6ef8184dee3 WatchSource:0}: Error finding container c480b9ea29f681f380919addbebb6abe5961412f2f0bc73b64b4f6ef8184dee3: Status 404 returned error can't find the container with id c480b9ea29f681f380919addbebb6abe5961412f2f0bc73b64b4f6ef8184dee3
Nov 03 16:31:31.886709 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.886671    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:31:31.886830 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.886760    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:31:31.886830 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:31.886803    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:31:32.070314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.070271    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" event={"ID":"ec52cc47-42d7-4e8f-b671-154086f71067","Type":"ContainerStarted","Data":"aa6cd6a05882d68412c576b1f1c582fd156a9b89244fa262d7dcda877e5577fd"}
Nov 03 16:31:32.070501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.070340    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" event={"ID":"ec52cc47-42d7-4e8f-b671-154086f71067","Type":"ContainerStarted","Data":"561681769af1b865a93847be0ce59d7c1fdce2a7da8a74f37f08f87d2845422d"}
Nov 03 16:31:32.072763 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.072683    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" event={"ID":"a16acfc1-e819-4a7c-b026-c7ffde68da12","Type":"ContainerStarted","Data":"8d717528f832c2631431b799c46b9008b12036123897867e269116d526e41c0c"}
Nov 03 16:31:32.072763 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.072732    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" event={"ID":"a16acfc1-e819-4a7c-b026-c7ffde68da12","Type":"ContainerStarted","Data":"c480b9ea29f681f380919addbebb6abe5961412f2f0bc73b64b4f6ef8184dee3"}
Nov 03 16:31:32.183773 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.183740    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:32.185788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.185755    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:32.185948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.185811    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:32.190009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.189944    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podStartSLOduration=2.189923597 podStartE2EDuration="2.189923597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:32.185510541 +0000 UTC m=+1253.764814426" watchObservedRunningTime="2026-11-03 16:31:32.189923597 +0000 UTC m=+1253.769227504"
Nov 03 16:31:32.319320 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.319269    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podStartSLOduration=2.319253057 podStartE2EDuration="2.319253057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:32.316135209 +0000 UTC m=+1253.895439084" watchObservedRunningTime="2026-11-03 16:31:32.319253057 +0000 UTC m=+1253.898556911"
Nov 03 16:31:32.499933 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.498539    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:32.509464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.504279    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:32.509464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:32.504334    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:33.184704 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:33.184661    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:33.185366 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:33.185333    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:33.509192 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:33.509100    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:33.509494 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:33.509462    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:34.207364 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:34.186453    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:34.207364 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:34.186518    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:34.497332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:34.497234    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:34.497592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:34.497564    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:35.086221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:35.085619    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:31:35.185219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:35.184855    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:35.185219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:35.185009    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:35.499467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:35.499432    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:35.521337 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:35.521292    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:36.063616 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:36.063564    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 16:31:36.187996 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:36.187954    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:36.188294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:36.188260    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:36.500224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:36.499758    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:36.500224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:36.499869    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:36.638228 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:36.636477    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:31:37.184235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:37.183622    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:37.184235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:37.183689    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:37.513280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:37.507815    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:37.513280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:37.507875    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:37.932481 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:31:37.932446    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:31:38.187075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:38.186584    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:38.187075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:38.186647    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:38.495489 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:38.495399    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:38.495489 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:38.495466    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:39.183648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:39.183608    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 16:31:39.184342 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:39.184299    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 16:31:39.496300 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:39.496212    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 16:31:39.496570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:39.496538    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 16:31:40.189929 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:40.189897    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:40.190458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:40.190431    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:40.191858 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:40.191837    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 16:31:40.507142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:40.507044    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:40.507685 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:40.507636    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:40.508941 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:40.508918    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:31:59.780604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.780520    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"]
Nov 03 16:31:59.786060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.786025    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873478    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873556    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgjn\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-kube-api-access-cqgjn\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873602    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873666    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873755    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873788    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873810    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873846    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.875240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.873896    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.974872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.974835    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.974983    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975007    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975025    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975048    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975079    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975225    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975273    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgjn\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-kube-api-access-cqgjn\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975317    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975616    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975645    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975807 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975680    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.975869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.975832    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.976275 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.976192    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.978892 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.978860    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.979472 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.979449    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:31:59.983825 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:31:59.983799    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"]
Nov 03 16:32:00.031982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.031903    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:32:00.033259 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.033229    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgjn\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-kube-api-access-cqgjn\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:32:00.103773 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.103731    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:32:00.532378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.532343    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"]
Nov 03 16:32:00.533060 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:32:00.533034    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9dd54f_d7e6_4a97_9ffc_45326e19d6ec.slice/crio-0e8be5ad1c84535f1ff2aebdbd62441151070ba76aff49f1e6daef42809cf081 WatchSource:0}: Error finding container 0e8be5ad1c84535f1ff2aebdbd62441151070ba76aff49f1e6daef42809cf081: Status 404 returned error can't find the container with id 0e8be5ad1c84535f1ff2aebdbd62441151070ba76aff49f1e6daef42809cf081
Nov 03 16:32:00.535549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.535516    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:00.535632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.535591    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:00.535632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.535624    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:00.693387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.693341    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" event={"ID":"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec","Type":"ContainerStarted","Data":"87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c"}
Nov 03 16:32:00.693624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.693416    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" event={"ID":"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec","Type":"ContainerStarted","Data":"0e8be5ad1c84535f1ff2aebdbd62441151070ba76aff49f1e6daef42809cf081"}
Nov 03 16:32:00.811317 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:00.811175    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podStartSLOduration=1.811159931 podStartE2EDuration="1.811159931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:00.808792165 +0000 UTC m=+1282.388096046" watchObservedRunningTime="2026-11-03 16:32:00.811159931 +0000 UTC m=+1282.390463785"
Nov 03 16:32:01.104285 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:01.104180    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:32:01.106347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:01.106316    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:32:01.106550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:01.106376    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:32:02.105185 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:02.105147    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:32:02.105710 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:02.105230    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:32:03.106018 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:03.105975    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:32:03.106502 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:03.106037    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:32:04.104301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:04.104263    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:32:04.104523 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:04.104322    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:32:05.104702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:05.104668    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:32:05.105125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:05.104734    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:32:06.104581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:06.104547    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:32:06.104799 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:06.104609    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:32:07.104814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:07.104762    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 16:32:07.105486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:07.105438    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 16:32:07.905255 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:32:07.905150    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:32:08.616371 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:08.616337    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:32:08.617783 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:08.617759    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:32:08.618668 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:08.618651    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:32:14.600852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.600813    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"]
Nov 03 16:32:14.606553 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.606528    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.740546 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.740502    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.740715 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.740551    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.740761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.740710    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.740805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.740777    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.740918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.740882    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.740988 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.740945    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.741074 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.741056    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.741138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.741099    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.741184 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.741138    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24vk\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-kube-api-access-f24vk\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.807705 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.807662    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"]
Nov 03 16:32:14.842812 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.842778    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.842828    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.842891    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.842955    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.843008    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.843046    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.843139    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.843167    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.843301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.843192    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24vk\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-kube-api-access-f24vk\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.844007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.843977    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.845708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.844719    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.845708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.845387    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.845870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.845778    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.846159 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.846135    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.846933 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.846907    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.847423 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.847402    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.904881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.904843    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"]
Nov 03 16:32:14.905740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.905713    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24vk\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-kube-api-access-f24vk\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.910173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.910150    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.912095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.912075    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:14.921274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.921250    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:14.977684 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:14.977647    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"]
Nov 03 16:32:15.046032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.045986    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046034    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046289 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046105    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681ca214-8021-47b1-b94f-73e47fea2010-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046289 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046141    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046289 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046179    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681ca214-8021-47b1-b94f-73e47fea2010-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046447 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046289    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdz6\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-kube-api-access-svdz6\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046447 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046414    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046539    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.046633 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.046572    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148393 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148354    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148577 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148471    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148577 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148500    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148577 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148528    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148732 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148666    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148732 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148711    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681ca214-8021-47b1-b94f-73e47fea2010-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148840 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148747    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148840 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148789    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681ca214-8021-47b1-b94f-73e47fea2010-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148845    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdz6\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-kube-api-access-svdz6\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.148946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148849    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-data\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.149067 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.148993    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-credential-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.149126 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.149103    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-certs\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.149352 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.149328    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-socket\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.149982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.149953    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681ca214-8021-47b1-b94f-73e47fea2010-istiod-ca-cert\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.151279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.151253    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-envoy\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.151410 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.151392    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681ca214-8021-47b1-b94f-73e47fea2010-istio-podinfo\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.204982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.204901    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-istio-token\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.205146 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.205072    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdz6\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-kube-api-access-svdz6\") pod \"e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") " pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.269561 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.269527    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:15.369482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.369445    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"]
Nov 03 16:32:15.413065 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.412995    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:15.413630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.413592    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:15.413880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.413855    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:15.656932 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:32:15.656898    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681ca214_8021_47b1_b94f_73e47fea2010.slice/crio-776ddc51c681dce3271a85920b714dccc319068a9cfdfc39a8155e53b7507a4a WatchSource:0}: Error finding container 776ddc51c681dce3271a85920b714dccc319068a9cfdfc39a8155e53b7507a4a: Status 404 returned error can't find the container with id 776ddc51c681dce3271a85920b714dccc319068a9cfdfc39a8155e53b7507a4a
Nov 03 16:32:15.657344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.657051    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"]
Nov 03 16:32:15.660109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.660069    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:15.660215 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.660157    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:15.660268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.660223    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:15.821615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.821580    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" event={"ID":"40eb21d3-2ad2-4c2c-a22c-7867a2352537","Type":"ContainerStarted","Data":"af29c24524df726c4ccf3a783c82bdb37a3119f988b28216dbb70e29aa44f15e"}
Nov 03 16:32:15.821788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.821640    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" event={"ID":"40eb21d3-2ad2-4c2c-a22c-7867a2352537","Type":"ContainerStarted","Data":"909d87a0abd0fdef6d071fde140ab21facab5f22acea36a46bc951330b601689"}
Nov 03 16:32:15.823981 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.823947    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" event={"ID":"681ca214-8021-47b1-b94f-73e47fea2010","Type":"ContainerStarted","Data":"b4f5ff099ca39e772ec7c533596b23bb2b3882b3b126f1e2dc03e1f9147d5075"}
Nov 03 16:32:15.824145 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.824005    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" event={"ID":"681ca214-8021-47b1-b94f-73e47fea2010","Type":"ContainerStarted","Data":"776ddc51c681dce3271a85920b714dccc319068a9cfdfc39a8155e53b7507a4a"}
Nov 03 16:32:15.921705 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.921671    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:15.946250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:15.946175    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podStartSLOduration=1.946157865 podStartE2EDuration="1.946157865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:15.942437636 +0000 UTC m=+1297.521741516" watchObservedRunningTime="2026-11-03 16:32:15.946157865 +0000 UTC m=+1297.525461788"
Nov 03 16:32:16.290227 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:16.277664    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:16.290227 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:16.277859    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 16:32:16.290227 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:16.277916    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 16:32:16.924363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:16.923894    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 16:32:16.924363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:16.923968    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 16:32:17.273141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:17.272596    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 16:32:17.273141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:17.272665    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 16:32:17.923225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:17.922635    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 16:32:17.923225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:17.922684    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 16:32:18.277283 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:18.276616    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 16:32:18.277283 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:18.276681    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 16:32:18.922542 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:18.922503    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 16:32:18.922860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:18.922816    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 16:32:19.272284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:19.270680    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 16:32:19.272284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:19.270793    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 16:32:19.923249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:19.922117    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 16:32:19.923249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:19.922190    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 16:32:20.275909 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:20.275814    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 16:32:20.276177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:20.276149    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 16:32:20.924343 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:20.923334    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 16:32:20.924343 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:20.923397    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 16:32:21.271479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:21.271384    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 16:32:21.271750 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:21.271719    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 16:32:21.922167 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:21.922123    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 16:32:21.922167 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:21.922194    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 16:32:22.270927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:22.270823    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 16:32:22.270927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:22.270896    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 16:32:23.016498 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:23.016381    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:23.016843 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:23.016819    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:23.017862 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:23.017834    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:32:23.182670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:23.182599    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podStartSLOduration=9.182579274 podStartE2EDuration="9.182579274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:16.069013614 +0000 UTC m=+1297.648317473" watchObservedRunningTime="2026-11-03 16:32:23.182579274 +0000 UTC m=+1304.761883130"
Nov 03 16:32:23.395786 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:23.395640    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:23.399091 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:23.399070    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:23.400041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:23.400027    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:32:24.463017 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.462978    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"]
Nov 03 16:32:24.468343 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.468319    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489008    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489062    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489089    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489174    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hjb\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-kube-api-access-k2hjb\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489228    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489399    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489460 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489435    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489486    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.489546 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.489533    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.519761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.519726    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-547a38-openshift-default-dockercfg-tnnkl\""
Nov 03 16:32:24.570430 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.570395    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"]
Nov 03 16:32:24.591418 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.591371    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592011    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592055    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592086    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hjb\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-kube-api-access-k2hjb\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592119    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592264    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592291    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592316    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.592451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.592364    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.595573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.594308    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.595573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.594335    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.595573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.594437    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.595573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.595514    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.595573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.595561    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.596473 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.596452    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.598378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.598357    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.649983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.649948    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.650308 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.650283    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hjb\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-kube-api-access-k2hjb\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:24.785470 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:24.785364    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:25.238074 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:25.238039    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"]
Nov 03 16:32:25.266598 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:32:25.266564    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e24cbe_7c8d_4e26_8bd5_1252fe33594d.slice/crio-7d50611f183ea31beaa3a70742b9c19a9df48807195871577bd0b701f01627d4 WatchSource:0}: Error finding container 7d50611f183ea31beaa3a70742b9c19a9df48807195871577bd0b701f01627d4: Status 404 returned error can't find the container with id 7d50611f183ea31beaa3a70742b9c19a9df48807195871577bd0b701f01627d4
Nov 03 16:32:25.269305 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:25.269273    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:25.269377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:25.269350    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:25.269419 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:25.269383    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:25.984498 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:25.984455    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" event={"ID":"93e24cbe-7c8d-4e26-8bd5-1252fe33594d","Type":"ContainerStarted","Data":"3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9"}
Nov 03 16:32:25.985142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:25.985115    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" event={"ID":"93e24cbe-7c8d-4e26-8bd5-1252fe33594d","Type":"ContainerStarted","Data":"7d50611f183ea31beaa3a70742b9c19a9df48807195871577bd0b701f01627d4"}
Nov 03 16:32:26.151414 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:26.151348    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podStartSLOduration=2.151330095 podStartE2EDuration="2.151330095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:26.149168816 +0000 UTC m=+1307.728472692" watchObservedRunningTime="2026-11-03 16:32:26.151330095 +0000 UTC m=+1307.730633950"
Nov 03 16:32:26.785994 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:26.785955    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:26.787865 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:26.787828    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:26.787983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:26.787895    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:27.790225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:27.789641    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:27.790225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:27.789747    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:28.793141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:28.792660    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:28.793141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:28.792731    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:29.669346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:29.669296    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 16:32:29.712376 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:29.712317    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:29.787970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:29.787931    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:29.788338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:29.788293    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:29.846697 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:29.845441    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:29.924740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:29.924621    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:30.017237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:30.013285    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:30.063512 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:30.063443    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:30.892223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:30.889320    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:30.916261 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:30.906340    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:30.916261 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:30.906489    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 16:32:30.934225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:30.932468    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:30.959224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:30.940979    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 16:32:31.095366 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.087075    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 16:32:31.122378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.111747    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:31.231225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.204880    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:31.327327 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.327262    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:31.447785 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.447724    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4v77c" podUID="32d8a9c3-50b8-4487-951a-0963519e47b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:31.553609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.553500    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:31.967632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.662843    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:31.984464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.984410    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:31.985158 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.985121    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 16:32:32.043257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.987450    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:32.044837 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:31.993552    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 16:32:32.152230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.148323    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:32.152230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.148383    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:32.334273 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.330414    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:32.362243 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:32:32.362181    5112 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.797s"
Nov 03 16:32:32.362745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.362728    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:32:32.363259 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.363240    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:32:32.363422 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.363411    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:32:32.363534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.363523    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"]
Nov 03 16:32:32.512618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.508476    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:32.641270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.637432    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.653654    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.653705    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"]
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.653764    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.653808    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.653906    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.654088    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.654127    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.654159    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.654192    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.654247    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.654279    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:32:32.660222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.654307    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:32:32.799530 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.799480    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.799849    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:32.800481 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800453    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800579 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800510    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800579 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800554    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800777 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800755    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800854 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800838    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800908 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800885    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800954 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800925    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.800991 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.800971    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqq5\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-kube-api-access-2wqq5\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.801100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.801072    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:32.902545 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902492    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.902759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902570    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.902759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902604    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.902759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902645    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqq5\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-kube-api-access-2wqq5\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.902759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902709    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.903003 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902900    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.903003 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902938    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.903003 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.902980    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.903167 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.903103    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.907015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.904737    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.907015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.905024    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-certs\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.907015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.905295    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-credential-socket\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.907015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.905915    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-data\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.907015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.906707    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istiod-ca-cert\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.911222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.910101    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-envoy\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.915253 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.914832    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-podinfo\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:32.997673 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:32.997629    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-token\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:33.005039 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.005004    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqq5\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-kube-api-access-2wqq5\") pod \"e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") " pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:33.301872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.301194    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:33.787104 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.787064    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 16:32:33.787235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.787135    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 16:32:33.790820 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:32:33.790792    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c48fd3_9c73_415d_aec3_a2a9fc42d2a7.slice/crio-6a649fb0e9a8661cdbbcc246c3f5a6443f7152aff2cfa62b1e939fc6b096f47b WatchSource:0}: Error finding container 6a649fb0e9a8661cdbbcc246c3f5a6443f7152aff2cfa62b1e939fc6b096f47b: Status 404 returned error can't find the container with id 6a649fb0e9a8661cdbbcc246c3f5a6443f7152aff2cfa62b1e939fc6b096f47b
Nov 03 16:32:33.791612 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.791582    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"]
Nov 03 16:32:33.794491 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.794425    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:33.794606 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.794530    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:33.794606 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:33.794578    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:34.349521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:34.349481    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" event={"ID":"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7","Type":"ContainerStarted","Data":"ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf"}
Nov 03 16:32:34.349955 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:34.349553    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" event={"ID":"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7","Type":"ContainerStarted","Data":"6a649fb0e9a8661cdbbcc246c3f5a6443f7152aff2cfa62b1e939fc6b096f47b"}
Nov 03 16:32:34.471630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:34.471564    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podStartSLOduration=3.47154282 podStartE2EDuration="3.47154282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:34.464500216 +0000 UTC m=+1316.043804072" watchObservedRunningTime="2026-11-03 16:32:34.47154282 +0000 UTC m=+1316.050846676"
Nov 03 16:32:35.301640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:35.301594    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:35.303394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:35.303358    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:32:35.303578 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:35.303419    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:32:35.791337 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:35.791291    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:32:35.792001 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:35.791956    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:32:36.303224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:36.302713    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:32:36.303224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:36.302794    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:32:36.700936 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:36.700892    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:36.703416 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:36.703392    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:36.705558 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:36.705457    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:32:37.301759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:37.301719    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:32:37.302173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:37.301778    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:32:37.918865 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:32:37.918823    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:32:38.301985 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:38.301882    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:32:38.301985 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:38.301957    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:32:39.302110 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:39.302065    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:32:39.302632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:39.302164    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:32:40.301565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:40.301528    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:32:40.301778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:40.301595    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:32:41.514595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:41.514552    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:41.515544 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:41.515496    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:41.516717 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:41.516696    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:32:45.517590 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.517551    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"]
Nov 03 16:32:45.534788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.534756    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.601229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.601179    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"]
Nov 03 16:32:45.636031 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.635993    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.636279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.636039    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.636279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.636091    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441034a0-b050-4421-8294-e6401e9049c4-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.636279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.636135    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.636279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.636160    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.636279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.636194    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.643736 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.643702    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.646331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.646300    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pb7\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-kube-api-access-d9pb7\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.646569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.646414    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441034a0-b050-4421-8294-e6401e9049c4-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.747618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.747586    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441034a0-b050-4421-8294-e6401e9049c4-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.747793 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.747750    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.747793 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.747771    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.747891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.747808    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441034a0-b050-4421-8294-e6401e9049c4-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.747927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.747902    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.747976 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.747936    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748103 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748078    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748210 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748124    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748210 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748140    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748187    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pb7\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-kube-api-access-d9pb7\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748256    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748347    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748481    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441034a0-b050-4421-8294-e6401e9049c4-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.748642 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.748558    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.750309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.750290    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.750555 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.750532    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441034a0-b050-4421-8294-e6401e9049c4-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.806326 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.806269    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.806448 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.806328    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pb7\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-kube-api-access-d9pb7\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:45.849813 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:45.849780    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:46.293892 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:32:46.293857    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441034a0_b050_4421_8294_e6401e9049c4.slice/crio-a71a12b7f689ca43e4fd628435b5789833a9295bc8ef5c86cc47f04d70d6df30 WatchSource:0}: Error finding container a71a12b7f689ca43e4fd628435b5789833a9295bc8ef5c86cc47f04d70d6df30: Status 404 returned error can't find the container with id a71a12b7f689ca43e4fd628435b5789833a9295bc8ef5c86cc47f04d70d6df30
Nov 03 16:32:46.294021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.293957    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"]
Nov 03 16:32:46.296729 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.296691    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:46.296839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.296760    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:46.296839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.296791    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:32:46.476385 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.476346    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" event={"ID":"441034a0-b050-4421-8294-e6401e9049c4","Type":"ContainerStarted","Data":"ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07"}
Nov 03 16:32:46.476559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.476412    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" event={"ID":"441034a0-b050-4421-8294-e6401e9049c4","Type":"ContainerStarted","Data":"a71a12b7f689ca43e4fd628435b5789833a9295bc8ef5c86cc47f04d70d6df30"}
Nov 03 16:32:46.600471 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.600369    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podStartSLOduration=1.600354384 podStartE2EDuration="1.600354384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:46.597139004 +0000 UTC m=+1328.176442859" watchObservedRunningTime="2026-11-03 16:32:46.600354384 +0000 UTC m=+1328.179658310"
Nov 03 16:32:46.850826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:46.850747    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:47.852276 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:47.852241    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:32:47.852684 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:47.852297    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:32:48.850717 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:48.850681    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:32:48.850889 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:48.850749    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:32:49.851024 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:49.850979    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:32:49.851559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:49.851056    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:32:50.850650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:50.850614    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:32:50.850854 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:50.850675    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:32:51.850984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:51.850941    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:32:51.851499 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:51.851020    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:32:51.977740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:51.977704    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"]
Nov 03 16:32:51.979713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:51.978137    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy" containerID="cri-o://ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3" gracePeriod=30
Nov 03 16:32:52.851003 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:52.850962    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 16:32:52.851484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:52.851047    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 16:32:53.855189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:53.855156    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:53.855710 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:53.855571    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:53.856355 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:53.856331    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:32:57.277351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.277325    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:32:57.353891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.353866    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-credential-socket\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354049 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.353906    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-certs\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354049 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.353946    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-envoy\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354049 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.353985    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-data\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354049 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354027    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-socket\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354071    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istiod-ca-cert\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354100    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-credential-socket" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:32:57.354338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354110    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-podinfo\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354273    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-token\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354294    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-certs" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:32:57.354338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354315    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-data" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:32:57.354338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354331    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-kube-api-access-lhgdq\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-kube-api-access-lhgdq\") pod \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\" (UID: \"e3f0bcb2-04c2-4831-9a63-066d3da3c93f\") "
Nov 03 16:32:57.354701 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354388    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-socket" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:32:57.354701 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354609    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istiod-ca-cert" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:32:57.354993 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354974    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.355061 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.354998    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.355061 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.355013    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.355061 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.355027    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.355061 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.355043    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.357525 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.357495    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-token" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:32:57.357638 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.357524    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-podinfo" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:32:57.357638 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.357610    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-envoy" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:32:57.357754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.357657    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-kube-api-access-lhgdq" pod "e3f0bcb2-04c2-4831-9a63-066d3da3c93f" (UID: "e3f0bcb2-04c2-4831-9a63-066d3da3c93f"). InnerVolumeSpecName "kube-api-access-lhgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:32:57.466401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.466367    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.466539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.466405    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.466539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.466420    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.466539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.466434    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgdq\" (UniqueName: \"kubernetes.io/projected/e3f0bcb2-04c2-4831-9a63-066d3da3c93f-kube-api-access-lhgdq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:32:57.573950 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.573864    5112 generic.go:356] "Generic (PLEG): container finished" podID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerID="ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3" exitCode=0
Nov 03 16:32:57.574111 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.574086    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" event={"ID":"e3f0bcb2-04c2-4831-9a63-066d3da3c93f","Type":"ContainerDied","Data":"ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3"}
Nov 03 16:32:57.574177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.574137    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw" event={"ID":"e3f0bcb2-04c2-4831-9a63-066d3da3c93f","Type":"ContainerDied","Data":"c89de53f775f489ebf7000c859c89b9c97e37ee4f8a750bb8ae5909c1ec4e8cf"}
Nov 03 16:32:57.574177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.574167    5112 scope.go:122] "RemoveContainer" containerID="ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3"
Nov 03 16:32:57.574375 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.574356    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"
Nov 03 16:32:57.594635 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.594614    5112 scope.go:122] "RemoveContainer" containerID="ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3"
Nov 03 16:32:57.594996 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:32:57.594971    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3\": container with ID starting with ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3 not found: ID does not exist" containerID="ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3"
Nov 03 16:32:57.595078 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.595006    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3"} err="failed to get container status \"ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3\": rpc error: code = NotFound desc = could not find container \"ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3\": container with ID starting with ad47c06792481b63a6248d8180bf50d9865631ed12c35da694e569d1b5b88ea3 not found: ID does not exist"
Nov 03 16:32:57.742390 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.742356    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"]
Nov 03 16:32:57.747880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:57.747849    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h8bxw"]
Nov 03 16:32:58.551458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:32:58.551426    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" path="/var/lib/kubelet/pods/e3f0bcb2-04c2-4831-9a63-066d3da3c93f/volumes"
Nov 03 16:33:07.089228 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:07.089164    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"]
Nov 03 16:33:07.090108 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:07.090045    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy" containerID="cri-o://8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015" gracePeriod=30
Nov 03 16:33:07.895727 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:33:07.895684    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:33:12.399939 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.399893    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:33:12.536666 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.536632    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istiod-ca-cert\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.536850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.536708    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-data\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.536905 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.536859    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-kube-api-access-dz69z\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-kube-api-access-dz69z\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.537005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.536985    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-socket\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.537095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537015    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-data" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:12.537095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537033    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istiod-ca-cert" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:33:12.537330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537281    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-socket" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:12.537422 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537352    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-credential-socket\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.537422 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537413    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-certs\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.537510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537461    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-envoy\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.537569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537507    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-token\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-token\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.537626 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537565    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-podinfo\") pod \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\" (UID: \"f8f69d9d-4c31-4e8e-acb2-f98fb3817236\") "
Nov 03 16:33:12.537626 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537586    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-credential-socket" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:12.537741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.537711    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-certs" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:12.538231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.538159    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.538231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.538183    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.538231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.538214    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.538231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.538227    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.538231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.538240    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.540425 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.540398    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-envoy" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:33:12.540553 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.540533    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-kube-api-access-dz69z" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "kube-api-access-dz69z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:12.540783 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.540757    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-podinfo" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:33:12.540877 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.540852    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-token" pod "f8f69d9d-4c31-4e8e-acb2-f98fb3817236" (UID: "f8f69d9d-4c31-4e8e-acb2-f98fb3817236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:33:12.640009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.639933    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.640009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.639959    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.640009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.639969    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.640009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.639980    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz69z\" (UniqueName: \"kubernetes.io/projected/f8f69d9d-4c31-4e8e-acb2-f98fb3817236-kube-api-access-dz69z\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:33:12.948531 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.948494    5112 generic.go:356] "Generic (PLEG): container finished" podID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerID="8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015" exitCode=0
Nov 03 16:33:12.948717 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.948547    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"
Nov 03 16:33:12.948717 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.948577    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" event={"ID":"f8f69d9d-4c31-4e8e-acb2-f98fb3817236","Type":"ContainerDied","Data":"8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015"}
Nov 03 16:33:12.948717 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.948635    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96" event={"ID":"f8f69d9d-4c31-4e8e-acb2-f98fb3817236","Type":"ContainerDied","Data":"a32171883d5cfc63b776c4c9e3ad11ec0c0781e8a644e1970a25c2c38153b7d9"}
Nov 03 16:33:12.948717 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.948668    5112 scope.go:122] "RemoveContainer" containerID="8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015"
Nov 03 16:33:12.959493 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.959458    5112 scope.go:122] "RemoveContainer" containerID="8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015"
Nov 03 16:33:12.959785 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:33:12.959765    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015\": container with ID starting with 8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015 not found: ID does not exist" containerID="8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015"
Nov 03 16:33:12.959862 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:12.959795    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015"} err="failed to get container status \"8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015\": rpc error: code = NotFound desc = could not find container \"8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015\": container with ID starting with 8721d4019c456507916925572923c9bb42a7467d842d677d54b045a92cd69015 not found: ID does not exist"
Nov 03 16:33:13.110911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:13.110874    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"]
Nov 03 16:33:13.158101 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:13.158069    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2cq96"]
Nov 03 16:33:14.551576 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:14.551537    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" path="/var/lib/kubelet/pods/f8f69d9d-4c31-4e8e-acb2-f98fb3817236/volumes"
Nov 03 16:33:24.995889 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:24.995685    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"]
Nov 03 16:33:24.996553 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:24.996529    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy"
Nov 03 16:33:24.996553 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:24.996552    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy"
Nov 03 16:33:24.996709 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:24.996585    5112 cpu_manager.go:414] "containerMap: removing container" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy"
Nov 03 16:33:24.996709 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:24.996591    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy"
Nov 03 16:33:24.996709 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:24.996707    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f0bcb2-04c2-4831-9a63-066d3da3c93f" containerName="istio-proxy"
Nov 03 16:33:24.996802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:24.996723    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f69d9d-4c31-4e8e-acb2-f98fb3817236" containerName="istio-proxy"
Nov 03 16:33:25.002412 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.002384    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.042238 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.042212    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"]
Nov 03 16:33:25.046848 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.046825    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.098249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.085925    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"]
Nov 03 16:33:25.114681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.114649    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.114831 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.114693    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319dc61-43c9-4409-9d22-b81507075236-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.114831 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.114798    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.114963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.114948    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.115067 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.115042    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdxv\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-kube-api-access-zcdxv\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.115121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.115082    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.115217 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.115184    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.115280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.115238    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.115332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.115307    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319dc61-43c9-4409-9d22-b81507075236-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.189947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.189900    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"]
Nov 03 16:33:25.216999 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.216960    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.217176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217018    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.217176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217102    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.217176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217140    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319dc61-43c9-4409-9d22-b81507075236-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.217708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217677    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.217835 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217757    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxvx\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-kube-api-access-cjxvx\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.217835 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217793    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.217965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217919    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.217965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217944    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319dc61-43c9-4409-9d22-b81507075236-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.218073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.217989    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.218073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.218025    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f67a9-7a18-4994-800c-66c7690aff43-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.218073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.218053    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f67a9-7a18-4994-800c-66c7690aff43-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.218278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.218142    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.218278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.218167    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.218639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.218615    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.218900 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.218879    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.219017 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.218991    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319dc61-43c9-4409-9d22-b81507075236-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.219237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.219212    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.219335 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.219297    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdxv\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-kube-api-access-zcdxv\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.219335 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.219330    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.219453 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.219359    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.219453 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.219383    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.220036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.220013    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.220659 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.220634    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319dc61-43c9-4409-9d22-b81507075236-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.221225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.221161    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.273816 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.273759    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.275175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.275153    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdxv\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-kube-api-access-zcdxv\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.318831 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.318806    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:25.320608 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.320587    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.320690 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.320631    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxvx\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-kube-api-access-cjxvx\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.320690 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.320651    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.320851 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.320827    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f67a9-7a18-4994-800c-66c7690aff43-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.320960 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.320855    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f67a9-7a18-4994-800c-66c7690aff43-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.320960 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.320933    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321013    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321044    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321055    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-data\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321089    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321425    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-credential-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321455    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-certs\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321809 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321650    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-socket\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.321874 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.321848    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f67a9-7a18-4994-800c-66c7690aff43-istiod-ca-cert\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.323591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.323093    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-envoy\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.323655 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.323599    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f67a9-7a18-4994-800c-66c7690aff43-istio-podinfo\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.380946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.380917    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-istio-token\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.382293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.382272    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxvx\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-kube-api-access-cjxvx\") pod \"e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") " pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.660951 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.660924    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:25.757669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.757634    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"]
Nov 03 16:33:25.762068 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.762023    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:25.762189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.762090    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:25.762189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:25.762128    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:26.047560 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.047533    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"]
Nov 03 16:33:26.049547 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:33:26.049512    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487f67a9_7a18_4994_800c_66c7690aff43.slice/crio-32b3b1884f9684927206d014bd3e9801ad9ccc0270b893f181dbcfa1266f52ab WatchSource:0}: Error finding container 32b3b1884f9684927206d014bd3e9801ad9ccc0270b893f181dbcfa1266f52ab: Status 404 returned error can't find the container with id 32b3b1884f9684927206d014bd3e9801ad9ccc0270b893f181dbcfa1266f52ab
Nov 03 16:33:26.052492 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.052443    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:26.052685 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.052548    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:26.052685 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.052598    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:26.060274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.060209    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" event={"ID":"c319dc61-43c9-4409-9d22-b81507075236","Type":"ContainerStarted","Data":"84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f"}
Nov 03 16:33:26.060383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.060281    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" event={"ID":"c319dc61-43c9-4409-9d22-b81507075236","Type":"ContainerStarted","Data":"536146626ec5d19a475b9822603644a1165c8c087e563bc334317714aa804a9c"}
Nov 03 16:33:26.179235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.179153    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podStartSLOduration=2.179134429 podStartE2EDuration="2.179134429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:26.177109968 +0000 UTC m=+1367.756413824" watchObservedRunningTime="2026-11-03 16:33:26.179134429 +0000 UTC m=+1367.758438284"
Nov 03 16:33:26.319451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.319352    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:26.321282 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.321252    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:33:26.321456 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:26.321315    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:33:27.128491 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.128454    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" event={"ID":"487f67a9-7a18-4994-800c-66c7690aff43","Type":"ContainerStarted","Data":"13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2"}
Nov 03 16:33:27.128976 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.128522    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" event={"ID":"487f67a9-7a18-4994-800c-66c7690aff43","Type":"ContainerStarted","Data":"32b3b1884f9684927206d014bd3e9801ad9ccc0270b893f181dbcfa1266f52ab"}
Nov 03 16:33:27.319614 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.319562    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:33:27.319800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.319627    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:33:27.353353 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.353297    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podStartSLOduration=3.353283496 podStartE2EDuration="3.353283496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:27.305759562 +0000 UTC m=+1368.885063420" watchObservedRunningTime="2026-11-03 16:33:27.353283496 +0000 UTC m=+1368.932587350"
Nov 03 16:33:27.681872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.681787    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:27.684262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.683907    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 16:33:27.684262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:27.683971    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 16:33:28.344550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:28.340251    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:33:28.344550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:28.340307    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:33:28.669221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:28.663500    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 16:33:28.669221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:28.663558    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 16:33:29.319681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:29.319637    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:33:29.319985 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:29.319957    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:33:29.670221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:29.666493    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 16:33:29.670221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:29.666563    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 16:33:30.320355 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.320316    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:33:30.320660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.320630    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:33:30.669664 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.669626    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 16:33:30.669976 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.669945    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 16:33:30.894224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.894006    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"]
Nov 03 16:33:30.923042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.922845    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.941583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.941551    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.941891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.941866    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.942151 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.942128    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.942383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.942363    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.942625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.942605    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lglh\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-kube-api-access-4lglh\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.942810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.942791    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.942992 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.942976    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.943158 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.943140    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:30.943355 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:30.943338    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.030932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.030892    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"]
Nov 03 16:33:31.045060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045032    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045263 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045169    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045263 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045238    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045390 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045316    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045390 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045365    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lglh\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-kube-api-access-4lglh\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045414    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045474    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045514    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045548    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045757 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045555    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-certs\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045757 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045568    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-data\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.045952 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.045927    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.046815 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.046789    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istiod-ca-cert\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.047001 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.046977    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-credential-socket\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.048804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.048777    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-podinfo\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.049419 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.049397    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-envoy\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.132477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.132437    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lglh\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-kube-api-access-4lglh\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.138357 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.138323    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-token\") pod \"e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") " pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.249550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.249436    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:31.333417 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.332509    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:33:31.333417 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.332572    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:33:31.681224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.667613    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 16:33:31.681224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.667674    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 16:33:31.833669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.833370    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"]
Nov 03 16:33:31.835033 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.834991    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:31.835153 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.835079    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:31.835153 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.835126    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:31.849898 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.849871    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"]
Nov 03 16:33:31.875615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.875595    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:31.942672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.937639    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"]
Nov 03 16:33:31.947783 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.947759    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"]
Nov 03 16:33:31.948021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:31.947995    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.023251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.022523    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"]
Nov 03 16:33:32.029667 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029637    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.029784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029684    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.029784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029741    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.029899 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029836    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.029899 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029886    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.030012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029909    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.030012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029940    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.030012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.029972    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.030163 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.030086    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.034563    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.034789    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.034826    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.034943    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrl4\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-kube-api-access-brrl4\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.035022    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.035052    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.035121    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.035313    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.035619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.035375    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxf8q\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-kube-api-access-dxf8q\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.137734 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.137670    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrl4\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-kube-api-access-brrl4\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.137940 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.137754    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.137940 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.137792    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.137940 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.137827    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.137940 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.137890    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.137940 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.137930    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxf8q\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-kube-api-access-dxf8q\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.138175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.137994    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.138175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138032    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.138175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138091    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.138175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138167    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138269    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138306    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138351    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138391    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138489    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138530    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138598    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.139138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.138635    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.140097 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.140064    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.144245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.143505    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.144245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.143814    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.144245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.144041    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.144547 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.144393    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.144547 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.144373    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-data\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.145607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.144897    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istiod-ca-cert\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.145607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.145237    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.145789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.145764    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-credential-socket\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.146663 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.146635    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-certs\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.148014 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.147984    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.149147 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.149113    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.151095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.150799    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-envoy\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.151095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.150898    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-podinfo\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.202959 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.202853    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.204615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.204564    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxf8q\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-kube-api-access-dxf8q\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.210606 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.210513    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrl4\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-kube-api-access-brrl4\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.211167 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.211125    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-token\") pod \"e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") " pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.319310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.319271    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 16:33:32.319479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.319336    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 16:33:32.325880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.325639    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:32.342518 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.342485    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:32.366741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.361418    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" event={"ID":"5c0f5855-bc6a-4677-93ce-dd659bf590d7","Type":"ContainerStarted","Data":"1773357816be7565e32e148df2759cc2de3a3ccb86a24608082c6a39cb003968"}
Nov 03 16:33:32.366741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.361527    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" event={"ID":"5c0f5855-bc6a-4677-93ce-dd659bf590d7","Type":"ContainerStarted","Data":"415ed70eecfb46ae5d65850f41d2f24e6d1b023226221478abb6e39cd78fd6c5"}
Nov 03 16:33:32.686264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.684365    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 16:33:32.686264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.684440    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 16:33:32.700374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:32.700316    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podStartSLOduration=2.700298247 podStartE2EDuration="2.700298247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:32.600853089 +0000 UTC m=+1374.180156992" watchObservedRunningTime="2026-11-03 16:33:32.700298247 +0000 UTC m=+1374.279602103"
Nov 03 16:33:33.015284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.014067    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"]
Nov 03 16:33:33.043299 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:33:33.043260    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f59c3b3_743c_4afe_8efe_dc5dc906a495.slice/crio-e71be1d3eea61c71e4a48b53c5aca6f4c759bb3d834304a41915069d9ebc6ce3 WatchSource:0}: Error finding container e71be1d3eea61c71e4a48b53c5aca6f4c759bb3d834304a41915069d9ebc6ce3: Status 404 returned error can't find the container with id e71be1d3eea61c71e4a48b53c5aca6f4c759bb3d834304a41915069d9ebc6ce3
Nov 03 16:33:33.046257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.046194    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:33.046357 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.046312    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:33.046398 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.046361    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:33.084223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.071873    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"]
Nov 03 16:33:33.128507 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.124933    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:33.128507 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.125020    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:33.128507 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.125067    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:33:33.277589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.265386    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:33.277589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.267141    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:33.277589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.267191    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:33.526736 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.500959    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" event={"ID":"f7d861a3-b90a-456f-9f54-ff31f6c09f1e","Type":"ContainerStarted","Data":"ee89b0a75c11466c420f7ad801aa248b7a679c039b96d2f36e5027d41ef16851"}
Nov 03 16:33:33.550297 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.528301    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" event={"ID":"6f59c3b3-743c-4afe-8efe-dc5dc906a495","Type":"ContainerStarted","Data":"e71be1d3eea61c71e4a48b53c5aca6f4c759bb3d834304a41915069d9ebc6ce3"}
Nov 03 16:33:33.743265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.742745    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 16:33:33.743265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:33.742795    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 16:33:34.255232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.255161    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:34.255703 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.255238    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:34.326904 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.326274    5112 patch_prober.go:29] interesting pod/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:33:34.326904 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.326335    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:34.441416 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.440038    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:34.442268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.442115    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:34.443179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.443156    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:33:34.593307 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.587708    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" event={"ID":"f7d861a3-b90a-456f-9f54-ff31f6c09f1e","Type":"ContainerStarted","Data":"97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e"}
Nov 03 16:33:34.593307 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.587774    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" event={"ID":"6f59c3b3-743c-4afe-8efe-dc5dc906a495","Type":"ContainerStarted","Data":"bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3"}
Nov 03 16:33:34.832357 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.828360    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podStartSLOduration=3.828337921 podStartE2EDuration="3.828337921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:34.822956816 +0000 UTC m=+1376.402260688" watchObservedRunningTime="2026-11-03 16:33:34.828337921 +0000 UTC m=+1376.407641777"
Nov 03 16:33:34.963222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.962753    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podStartSLOduration=3.962730137 podStartE2EDuration="3.962730137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:34.947710392 +0000 UTC m=+1376.527014282" watchObservedRunningTime="2026-11-03 16:33:34.962730137 +0000 UTC m=+1376.542033992"
Nov 03 16:33:34.970226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.969081    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:34.975273 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:34.973245    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:35.015826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.015452    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:33:35.293292 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.292343    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:35.293292 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.292406    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:35.364247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.353297    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:35.364247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.355324    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:35.364247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.355376    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:35.364247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.357998    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:35.364247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.358098    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:35.364247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:35.358137    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:36.358569 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.337853    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:36.388266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.337864    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:36.446222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.433475    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:36.446222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.434392    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:36.446222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.434427    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:36.446222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.435418    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:36.618282 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.614470    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"]
Nov 03 16:33:36.908271 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.894395    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.009224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:36.982211    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9rq\" (UniqueName: \"kubernetes.io/projected/66613f0e-46a9-4637-abde-6edbb19354f2-kube-api-access-xf9rq\") pod \"payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.009805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.009773    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66613f0e-46a9-4637-abde-6edbb19354f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.010056 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.008877    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e998bdb4-dockercfg-nbntz\""
Nov 03 16:33:37.010161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.008823    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e998bdb4\""
Nov 03 16:33:37.026311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.026289    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"]
Nov 03 16:33:37.134223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.111301    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66613f0e-46a9-4637-abde-6edbb19354f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.134627 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.134596    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9rq\" (UniqueName: \"kubernetes.io/projected/66613f0e-46a9-4637-abde-6edbb19354f2-kube-api-access-xf9rq\") pod \"payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.139303 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.120060    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66613f0e-46a9-4637-abde-6edbb19354f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.238018 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.237909    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9rq\" (UniqueName: \"kubernetes.io/projected/66613f0e-46a9-4637-abde-6edbb19354f2-kube-api-access-xf9rq\") pod \"payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.252223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.250858    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:37.252223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.250913    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:37.345613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.338344    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:37.345613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.338395    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:37.345613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.338566    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:37.345613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.344128    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:37.345613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:37.344169    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:38.135385 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:33:38.113467    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:33:38.322372 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:38.314300    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:38.322661 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:38.322621    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:38.345385 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:38.345345    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:38.345667 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:38.345641    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:38.346788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:38.346765    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:38.346959 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:38.346940    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:39.013222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.012810    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:39.033946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.033795    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"]
Nov 03 16:33:39.253387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.251324    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:39.253387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.251395    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:39.337704 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.336297    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:39.337704 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.336357    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:39.361298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.353399    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:39.361298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.353464    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:39.852390 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.852347    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t" event={"ID":"66613f0e-46a9-4637-abde-6edbb19354f2","Type":"ContainerStarted","Data":"14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c"}
Nov 03 16:33:39.852600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.852442    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t" event={"ID":"66613f0e-46a9-4637-abde-6edbb19354f2","Type":"ContainerStarted","Data":"5367e2e503ca9bc0a9b75586bdd0f29cbf830fc8fae352c1ec53210a9b990423"}
Nov 03 16:33:39.853106 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:39.853083    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:40.250461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:40.250414    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 16:33:40.250677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:40.250482    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 16:33:40.330249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:40.329450    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:40.330249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:40.329496    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:40.343033 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:40.343001    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:40.343222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:40.343045    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:41.348223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:41.339568    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:41.348223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:41.339632    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:41.361722 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:41.360837    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:41.361722 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:41.360909    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:42.083513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.083459    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:42.336063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.335965    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:42.336702 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.336680    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:33:42.336916 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.336881    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:42.337484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.336632    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:42.359671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.359636    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:42.360305 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.360258    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:42.635274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.634285    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:42.690219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.667467    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:42.690219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.668883    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:42.690219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.677706    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:33:42.725789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.725746    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:42.890280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:42.889572    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t" podStartSLOduration=6.889551201 podStartE2EDuration="6.889551201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:39.991381109 +0000 UTC m=+1381.570684965" watchObservedRunningTime="2026-11-03 16:33:42.889551201 +0000 UTC m=+1384.468855059"
Nov 03 16:33:43.084224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.083319    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:43.084224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.084036    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:43.111233 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.096289    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:33:43.379222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.357819    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 16:33:43.505128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.505084    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 16:33:43.698662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.478296    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:43.698662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.446792    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 16:33:43.698662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.552960    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 16:33:43.722224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.703794    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 16:33:43.796254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.796161    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 16:33:43.824228 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:43.814772    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:33:44.254226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:44.253749    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:44.381224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:44.380262    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded"
Nov 03 16:33:44.581344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:44.581238    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:44.765395 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:44.765339    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:44.766050 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:44.766016    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 16:33:44.776221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:44.768727    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:44.805322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:44.805275    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 16:33:45.132314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.116828    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:33:45.132632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.132605    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:33:45.265223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.261224    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:45.471351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.471301    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:45.524769 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.508409    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:33:45.524769 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.508469    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:45.526298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.525617    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:33:45.526298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.525677    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:45.598216 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.598063    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 03 16:33:45.798282 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.773058    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:45.798754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.783956    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:45.810230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.799121    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:33:45.810632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.784493    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:45.818352 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:45.818324    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:33:46.584535 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.572354    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 16:33:46.619219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.618505    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:46.810570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.810520    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-b9ppx" podUID="9b88bbf9-f49b-4e54-9eea-900cc0e5e078" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.57:50052\" within 1s: context deadline exceeded"
Nov 03 16:33:47.078221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.810904    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:33:47.217211 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.820305    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 03 16:33:47.294526 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.846648    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded"
Nov 03 16:33:47.299367 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.847263    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 16:33:47.501224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.847312    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 03 16:33:47.501442 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:46.907539    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:33:47.501442 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:47.501413    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:47.560970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:47.560929    5112 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded"
Nov 03 16:33:47.856280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:47.588321    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:47.856280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:47.751541    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:47.856280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:47.763354    5112 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-s557v container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 16:33:47.856280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:47.856190    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-s557v" podUID="f2972ac4-c9e1-45be-ba6d-0b5199d445a7" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:47.857003 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:47.807422    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 16:33:48.243223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.229101    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:48.334948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.334893    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:48.466828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.466782    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w" podUID="105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 16:33:48.491928 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.491406    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:48.562289 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.561436    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Nov 03 16:33:48.742748 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:33:48.742702    5112 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.673s"
Nov 03 16:33:48.745264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.745231    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:48.745482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.745454    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:48.747987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.747972    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:48.750384 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.750369    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:48.750593 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.750578    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:33:48.918389 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.916334    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:48.918389 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.916618    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 16:33:48.918389 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:48.916900    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 16:33:49.031288 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.013962    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"]
Nov 03 16:33:49.209811 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.209715    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:33:49.211369 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.211253    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.212553 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.211387    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:33:49.212771 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.212757    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"]
Nov 03 16:33:49.212928 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.212916    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"]
Nov 03 16:33:49.288241 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.272955    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a261207d-43f5-46e5-9b37-80da100d2abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.288241 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.273022    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h2n\" (UniqueName: \"kubernetes.io/projected/a261207d-43f5-46e5-9b37-80da100d2abb-kube-api-access-n6h2n\") pod \"payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.334630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.334451    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"]
Nov 03 16:33:49.334960 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.334936    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:49.343212 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.338413    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:33:49.364224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.355490    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b774106-dockercfg-2twrc\""
Nov 03 16:33:49.371220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.364527    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b774106\""
Nov 03 16:33:49.387987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.367700    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:33:49.416218 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.386066    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a261207d-43f5-46e5-9b37-80da100d2abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.416574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.416551    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h2n\" (UniqueName: \"kubernetes.io/projected/a261207d-43f5-46e5-9b37-80da100d2abb-kube-api-access-n6h2n\") pod \"payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.418264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.387026    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a261207d-43f5-46e5-9b37-80da100d2abb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.477996 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.477915    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 16:33:49.523223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.519353    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrw5\" (UniqueName: \"kubernetes.io/projected/2346e75a-c070-4479-a39c-f074d22c38c5-kube-api-access-vsrw5\") pod \"payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") " pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:49.523223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.519410    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2346e75a-c070-4479-a39c-f074d22c38c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") " pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:49.539270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.539242    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 16:33:49.541665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.541632    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 16:33:49.544428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.544394    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h2n\" (UniqueName: \"kubernetes.io/projected/a261207d-43f5-46e5-9b37-80da100d2abb-kube-api-access-n6h2n\") pod \"payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.546159 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.546133    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:49.547935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.547917    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 16:33:49.692589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.692482    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:33:49.698889 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.698854    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrw5\" (UniqueName: \"kubernetes.io/projected/2346e75a-c070-4479-a39c-f074d22c38c5-kube-api-access-vsrw5\") pod \"payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") " pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:49.700780 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.700753    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2346e75a-c070-4479-a39c-f074d22c38c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") " pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:49.702545 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.701794    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2346e75a-c070-4479-a39c-f074d22c38c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") " pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:49.761335 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.760997    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrw5\" (UniqueName: \"kubernetes.io/projected/2346e75a-c070-4479-a39c-f074d22c38c5-kube-api-access-vsrw5\") pod \"payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") " pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:49.909333 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:49.909042    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:50.068434 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:33:50.068376    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda261207d_43f5_46e5_9b37_80da100d2abb.slice/crio-a1253f1ee55414df6eaa01ac5a1fd77788a003812f94a6ae631d8a808939a7aa WatchSource:0}: Error finding container a1253f1ee55414df6eaa01ac5a1fd77788a003812f94a6ae631d8a808939a7aa: Status 404 returned error can't find the container with id a1253f1ee55414df6eaa01ac5a1fd77788a003812f94a6ae631d8a808939a7aa
Nov 03 16:33:50.097053 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.097024    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"]
Nov 03 16:33:50.311492 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.311462    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"]
Nov 03 16:33:50.312677 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:33:50.312633    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2346e75a_c070_4479_a39c_f074d22c38c5.slice/crio-bb3520acba922669734e914a77357107e3f08e082edeba2b3716bb3fc4e0b303 WatchSource:0}: Error finding container bb3520acba922669734e914a77357107e3f08e082edeba2b3716bb3fc4e0b303: Status 404 returned error can't find the container with id bb3520acba922669734e914a77357107e3f08e082edeba2b3716bb3fc4e0b303
Nov 03 16:33:50.717176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.717132    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25" event={"ID":"2346e75a-c070-4479-a39c-f074d22c38c5","Type":"ContainerStarted","Data":"afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13"}
Nov 03 16:33:50.717377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.717217    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:33:50.717377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.717245    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25" event={"ID":"2346e75a-c070-4479-a39c-f074d22c38c5","Type":"ContainerStarted","Data":"bb3520acba922669734e914a77357107e3f08e082edeba2b3716bb3fc4e0b303"}
Nov 03 16:33:50.719107 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.719075    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2" event={"ID":"a261207d-43f5-46e5-9b37-80da100d2abb","Type":"ContainerStarted","Data":"7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23"}
Nov 03 16:33:50.719288 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.719126    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2" event={"ID":"a261207d-43f5-46e5-9b37-80da100d2abb","Type":"ContainerStarted","Data":"a1253f1ee55414df6eaa01ac5a1fd77788a003812f94a6ae631d8a808939a7aa"}
Nov 03 16:33:50.719288 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.719192    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:33:50.906329 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.906299    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:33:50.966366 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:50.966312    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2" podStartSLOduration=5.966297749 podStartE2EDuration="5.966297749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:50.962932495 +0000 UTC m=+1392.542236351" watchObservedRunningTime="2026-11-03 16:33:50.966297749 +0000 UTC m=+1392.545601605"
Nov 03 16:33:51.078538 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:51.078425    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25" podStartSLOduration=5.078410502 podStartE2EDuration="5.078410502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:51.077786505 +0000 UTC m=+1392.657090380" watchObservedRunningTime="2026-11-03 16:33:51.078410502 +0000 UTC m=+1392.657714356"
Nov 03 16:33:56.900828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:56.900794    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"]
Nov 03 16:33:56.905629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:56.905607    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.038403 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.037256    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da5d843-1527-45c5-b526-2f73c8512552-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") " pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.038403 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.037466    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgrps\" (UniqueName: \"kubernetes.io/projected/6da5d843-1527-45c5-b526-2f73c8512552-kube-api-access-lgrps\") pod \"payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") " pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.038403 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.037650    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"]
Nov 03 16:33:57.046744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.045805    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e6def865\""
Nov 03 16:33:57.046744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.045962    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e6def865-dockercfg-xmrz2\""
Nov 03 16:33:57.092109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.092064    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"]
Nov 03 16:33:57.098009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.097987    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.131036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.131011    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"]
Nov 03 16:33:57.139552 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.139518    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgrps\" (UniqueName: \"kubernetes.io/projected/6da5d843-1527-45c5-b526-2f73c8512552-kube-api-access-lgrps\") pod \"payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") " pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.139795 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.139772    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da5d843-1527-45c5-b526-2f73c8512552-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") " pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.140787 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.140761    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da5d843-1527-45c5-b526-2f73c8512552-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") " pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.198684 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.198606    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgrps\" (UniqueName: \"kubernetes.io/projected/6da5d843-1527-45c5-b526-2f73c8512552-kube-api-access-lgrps\") pod \"payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") " pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.223171 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.223143    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.241628 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.241600    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkxr6\" (UniqueName: \"kubernetes.io/projected/444e6657-e43a-4738-aa04-1c78dee2db86-kube-api-access-dkxr6\") pod \"payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.241741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.241726    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444e6657-e43a-4738-aa04-1c78dee2db86-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.343833 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.342886    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkxr6\" (UniqueName: \"kubernetes.io/projected/444e6657-e43a-4738-aa04-1c78dee2db86-kube-api-access-dkxr6\") pod \"payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.343833 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.342989    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444e6657-e43a-4738-aa04-1c78dee2db86-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.343833 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.343787    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444e6657-e43a-4738-aa04-1c78dee2db86-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.399673 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.399641    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkxr6\" (UniqueName: \"kubernetes.io/projected/444e6657-e43a-4738-aa04-1c78dee2db86-kube-api-access-dkxr6\") pod \"payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.414390 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.414365    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:57.604658 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.604630    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"]
Nov 03 16:33:57.606163 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:33:57.606134    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da5d843_1527_45c5_b526_2f73c8512552.slice/crio-9a27576f2e970d7495a1ce63839f4a18fabe8d37713157891a820fad9e3a2862 WatchSource:0}: Error finding container 9a27576f2e970d7495a1ce63839f4a18fabe8d37713157891a820fad9e3a2862: Status 404 returned error can't find the container with id 9a27576f2e970d7495a1ce63839f4a18fabe8d37713157891a820fad9e3a2862
Nov 03 16:33:57.789334 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.789230    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4" event={"ID":"6da5d843-1527-45c5-b526-2f73c8512552","Type":"ContainerStarted","Data":"851e67c1b912e9cb667a004b662b7e55a8427789e5f9eac2a21a941c0133f0c0"}
Nov 03 16:33:57.789334 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.789285    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4" event={"ID":"6da5d843-1527-45c5-b526-2f73c8512552","Type":"ContainerStarted","Data":"9a27576f2e970d7495a1ce63839f4a18fabe8d37713157891a820fad9e3a2862"}
Nov 03 16:33:57.789565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.789364    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:33:57.798307 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:33:57.798271    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444e6657_e43a_4738_aa04_1c78dee2db86.slice/crio-22b59581db891674f296353dad65d9929071f781adcc42e9d712658e65224d7b WatchSource:0}: Error finding container 22b59581db891674f296353dad65d9929071f781adcc42e9d712658e65224d7b: Status 404 returned error can't find the container with id 22b59581db891674f296353dad65d9929071f781adcc42e9d712658e65224d7b
Nov 03 16:33:57.798307 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.798293    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"]
Nov 03 16:33:57.904261 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:57.904190    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4" podStartSLOduration=1.904174132 podStartE2EDuration="1.904174132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:57.901148367 +0000 UTC m=+1399.480452244" watchObservedRunningTime="2026-11-03 16:33:57.904174132 +0000 UTC m=+1399.483477986"
Nov 03 16:33:58.800279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:58.800233    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9" event={"ID":"444e6657-e43a-4738-aa04-1c78dee2db86","Type":"ContainerStarted","Data":"5e951d0d44ff990854207d1314875418676d57b166cf2dd2ee5eaa228db3514f"}
Nov 03 16:33:58.800478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:58.800335    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9" event={"ID":"444e6657-e43a-4738-aa04-1c78dee2db86","Type":"ContainerStarted","Data":"22b59581db891674f296353dad65d9929071f781adcc42e9d712658e65224d7b"}
Nov 03 16:33:58.800478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:58.800364    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:33:58.914836 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:33:58.914778    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9" podStartSLOduration=1.914764918 podStartE2EDuration="1.914764918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:58.912334675 +0000 UTC m=+1400.491638531" watchObservedRunningTime="2026-11-03 16:33:58.914764918 +0000 UTC m=+1400.494068773"
Nov 03 16:34:01.730755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:01.730724    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:34:01.731237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:01.730929    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:34:01.920383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:01.920345    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"]
Nov 03 16:34:01.926469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:01.926445    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:01.988864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:01.979028    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-06f127ec\""
Nov 03 16:34:01.988864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:01.979250    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-06f127ec-dockercfg-dk8rw\""
Nov 03 16:34:01.988864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:01.986627    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"]
Nov 03 16:34:02.022879 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.022843    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrzx5\" (UniqueName: \"kubernetes.io/projected/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-kube-api-access-wrzx5\") pod \"payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.023157 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.023131    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.040759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.040725    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"]
Nov 03 16:34:02.047128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.047101    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.097218 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.097166    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"]
Nov 03 16:34:02.133625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.125235    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.140256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.137534    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5978ac-f897-4bfd-b3fd-60a9847f633b-plugins-config-volume\") pod \"payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") " pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.140256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.129530    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.140256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.137784    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9pf\" (UniqueName: \"kubernetes.io/projected/aa5978ac-f897-4bfd-b3fd-60a9847f633b-kube-api-access-sr9pf\") pod \"payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") " pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.140256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.137863    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrzx5\" (UniqueName: \"kubernetes.io/projected/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-kube-api-access-wrzx5\") pod \"payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.216219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.215349    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrzx5\" (UniqueName: \"kubernetes.io/projected/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-kube-api-access-wrzx5\") pod \"payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.242180 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.242084    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5978ac-f897-4bfd-b3fd-60a9847f633b-plugins-config-volume\") pod \"payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") " pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.242388 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.242294    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9pf\" (UniqueName: \"kubernetes.io/projected/aa5978ac-f897-4bfd-b3fd-60a9847f633b-kube-api-access-sr9pf\") pod \"payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") " pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.242598 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.242582    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.242918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.242899    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5978ac-f897-4bfd-b3fd-60a9847f633b-plugins-config-volume\") pod \"payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") " pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.299591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.299560    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9pf\" (UniqueName: \"kubernetes.io/projected/aa5978ac-f897-4bfd-b3fd-60a9847f633b-kube-api-access-sr9pf\") pod \"payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") " pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.365687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.365461    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:02.646866 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.646841    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"]
Nov 03 16:34:02.646967 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:34:02.646938    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212091e5_3ae7_4d1a_9510_5d0a0b654f6f.slice/crio-3d0d1cc9da5b79d1c496afabb19f725bc0eef385ae99da0dee08e64dd6a8d5c9 WatchSource:0}: Error finding container 3d0d1cc9da5b79d1c496afabb19f725bc0eef385ae99da0dee08e64dd6a8d5c9: Status 404 returned error can't find the container with id 3d0d1cc9da5b79d1c496afabb19f725bc0eef385ae99da0dee08e64dd6a8d5c9
Nov 03 16:34:02.806015 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:34:02.805981    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5978ac_f897_4bfd_b3fd_60a9847f633b.slice/crio-edff6266047874485cc34cd2ce93cf2a671faf88ddc21da25765283bd7492bd4 WatchSource:0}: Error finding container edff6266047874485cc34cd2ce93cf2a671faf88ddc21da25765283bd7492bd4: Status 404 returned error can't find the container with id edff6266047874485cc34cd2ce93cf2a671faf88ddc21da25765283bd7492bd4
Nov 03 16:34:02.806436 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.806294    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"]
Nov 03 16:34:02.835602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.835575    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98" event={"ID":"aa5978ac-f897-4bfd-b3fd-60a9847f633b","Type":"ContainerStarted","Data":"edff6266047874485cc34cd2ce93cf2a671faf88ddc21da25765283bd7492bd4"}
Nov 03 16:34:02.837137 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.837113    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh" event={"ID":"212091e5-3ae7-4d1a-9510-5d0a0b654f6f","Type":"ContainerStarted","Data":"bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6"}
Nov 03 16:34:02.837250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.837157    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh" event={"ID":"212091e5-3ae7-4d1a-9510-5d0a0b654f6f","Type":"ContainerStarted","Data":"3d0d1cc9da5b79d1c496afabb19f725bc0eef385ae99da0dee08e64dd6a8d5c9"}
Nov 03 16:34:02.837250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.837180    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:02.950272 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:02.950135    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh" podStartSLOduration=1.950117257 podStartE2EDuration="1.950117257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:02.949530158 +0000 UTC m=+1404.528834025" watchObservedRunningTime="2026-11-03 16:34:02.950117257 +0000 UTC m=+1404.529421111"
Nov 03 16:34:03.846805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:03.846766    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98" event={"ID":"aa5978ac-f897-4bfd-b3fd-60a9847f633b","Type":"ContainerStarted","Data":"44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac"}
Nov 03 16:34:03.847230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:03.846842    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:03.961257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:03.961185    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98" podStartSLOduration=2.961173441 podStartE2EDuration="2.961173441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:03.960259142 +0000 UTC m=+1405.539563022" watchObservedRunningTime="2026-11-03 16:34:03.961173441 +0000 UTC m=+1405.540477295"
Nov 03 16:34:07.896263 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:34:07.896189    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:34:08.803810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:08.803778    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:34:09.813020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:09.812990    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:34:12.679825 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.679789    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"]
Nov 03 16:34:12.687186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.687160    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.740490 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.740464    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 16:34:12.786326 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.786294    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 16:34:12.791090 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.791012    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-57mjv\""
Nov 03 16:34:12.792222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.791547    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 16:34:12.821046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.820983    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"]
Nov 03 16:34:12.840431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.840398    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.840615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.840436    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.840615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.840467    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.840744 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.840709    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbe652f-e7ae-4938-bff9-c46e098caa35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.840800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.840761    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.840800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.840788    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.841017 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.840996    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ngm\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-kube-api-access-v6ngm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.942394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.942311    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbe652f-e7ae-4938-bff9-c46e098caa35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.942394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.942354    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.942394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.942376    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.942625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.942588    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ngm\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-kube-api-access-v6ngm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.942683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.942668    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.942729 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.942701    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.942781 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.942733    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.943324 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.943292    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.946688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.946661    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.946808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.946682    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbe652f-e7ae-4938-bff9-c46e098caa35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.946808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.946746    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.946808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.946754    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.999081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.999051    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:12.999271 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:12.999176    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ngm\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-kube-api-access-v6ngm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-km6lh\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:13.004011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:13.003990    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:13.436470 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:34:13.436439    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbe652f_e7ae_4938_bff9_c46e098caa35.slice/crio-3c45667da4e9267a1d643c91fd4610880140a32943f88f3f379dc3c10f5999fa WatchSource:0}: Error finding container 3c45667da4e9267a1d643c91fd4610880140a32943f88f3f379dc3c10f5999fa: Status 404 returned error can't find the container with id 3c45667da4e9267a1d643c91fd4610880140a32943f88f3f379dc3c10f5999fa
Nov 03 16:34:13.436593 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:13.436523    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"]
Nov 03 16:34:13.850757 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:13.850673    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:34:13.946384 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:13.946346    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh" event={"ID":"3fbe652f-e7ae-4938-bff9-c46e098caa35","Type":"ContainerStarted","Data":"3c45667da4e9267a1d643c91fd4610880140a32943f88f3f379dc3c10f5999fa"}
Nov 03 16:34:14.941231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:14.940643    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:34:15.817225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:15.815636    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"]
Nov 03 16:34:16.183403 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.174732    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"]
Nov 03 16:34:16.183403 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.177382    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.291884    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.291973    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.292012    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.292047    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.292124    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2xh\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-kube-api-access-hr2xh\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.292165    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.292190    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.292234    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e248dc-34a8-4b70-9e9b-660296a68e80-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.292809 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.292374    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.405224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.401337    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.410218 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.405437    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.410414 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.410389    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e248dc-34a8-4b70-9e9b-660296a68e80-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.442309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.412507    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.515231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.494492    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-certs\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.583302 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.516835    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.583668 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.583639    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.583932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.583911    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.584097 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.584079    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.584268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.584246    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.584506 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.584484    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2xh\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-kube-api-access-hr2xh\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.633456 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.590622    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-data\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.644300 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.570759    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e248dc-34a8-4b70-9e9b-660296a68e80-istiod-ca-cert\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.668314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.596688    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-credential-socket\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.914291 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.904250    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-envoy\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:16.914469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:16.914370    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-token\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:17.126291 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.126257    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2xh\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-kube-api-access-hr2xh\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:17.221505 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.221415    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-podinfo\") pod \"e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") " pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:17.234915 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.234867    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 03 16:34:17.352234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.352122    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 03 16:34:17.489327 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.489168    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:34:17.490783 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.490747    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:17.494849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.494814    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:34:17.549941 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:34:17.546152    5112 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.002s"
Nov 03 16:34:17.550313 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.550288    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"]
Nov 03 16:34:17.564293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.564262    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"]
Nov 03 16:34:17.564822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.564709    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.640121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.640050    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ls66\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-kube-api-access-8ls66\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.642224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.640460    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.642463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.642440    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.644085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.642609    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.647488 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.644366    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.647721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.647695    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.648030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.648009    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.648152 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.648136    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.648435 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.648415    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750239    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750524 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750320    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ls66\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-kube-api-access-8ls66\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750524 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750403    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750524 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750421    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750524 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750460    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750753 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750533    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750753 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750555    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750753 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750663    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.750753 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.750681    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.751070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.751032    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.752316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.751250    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.752531 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.751484    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.752649 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.751748    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.752754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.752560    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.753655 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.753632    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.762990 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.762964    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.812893 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.812152    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.812893 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.812389    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ls66\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-kube-api-access-8ls66\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:17.920143 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:17.920101    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:18.048054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.047711    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"]
Nov 03 16:34:18.052035 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:34:18.052000    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e248dc_34a8_4b70_9e9b_660296a68e80.slice/crio-57db5c65171adabb4359934d6a58f570f6dc7946e9a3cda5d387fae7a255d70e WatchSource:0}: Error finding container 57db5c65171adabb4359934d6a58f570f6dc7946e9a3cda5d387fae7a255d70e: Status 404 returned error can't find the container with id 57db5c65171adabb4359934d6a58f570f6dc7946e9a3cda5d387fae7a255d70e
Nov 03 16:34:18.055339 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.055281    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:18.055482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.055377    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:18.055482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.055434    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:18.329293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.329191    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"]
Nov 03 16:34:18.343543 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:34:18.343484    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca07c6b_d01a_403d_81d6_2dbebcbd8c13.slice/crio-7580d11fda6d12f685547298d99ba95c36f5146412d82765134d3019540c3278 WatchSource:0}: Error finding container 7580d11fda6d12f685547298d99ba95c36f5146412d82765134d3019540c3278: Status 404 returned error can't find the container with id 7580d11fda6d12f685547298d99ba95c36f5146412d82765134d3019540c3278
Nov 03 16:34:18.346802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.346766    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:18.346925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.346854    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:18.346925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.346903    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:18.589453 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.589373    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" event={"ID":"d0e248dc-34a8-4b70-9e9b-660296a68e80","Type":"ContainerStarted","Data":"7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb"}
Nov 03 16:34:18.589698 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.589663    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" event={"ID":"d0e248dc-34a8-4b70-9e9b-660296a68e80","Type":"ContainerStarted","Data":"57db5c65171adabb4359934d6a58f570f6dc7946e9a3cda5d387fae7a255d70e"}
Nov 03 16:34:18.591165 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.591141    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" event={"ID":"dca07c6b-d01a-403d-81d6-2dbebcbd8c13","Type":"ContainerStarted","Data":"57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b"}
Nov 03 16:34:18.591298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.591184    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" event={"ID":"dca07c6b-d01a-403d-81d6-2dbebcbd8c13","Type":"ContainerStarted","Data":"7580d11fda6d12f685547298d99ba95c36f5146412d82765134d3019540c3278"}
Nov 03 16:34:18.712660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.712602    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podStartSLOduration=3.712582282 podStartE2EDuration="3.712582282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:18.70796808 +0000 UTC m=+1420.287271967" watchObservedRunningTime="2026-11-03 16:34:18.712582282 +0000 UTC m=+1420.291886139"
Nov 03 16:34:18.825070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.825016    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podStartSLOduration=2.825003501 podStartE2EDuration="2.825003501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:18.824830902 +0000 UTC m=+1420.404134794" watchObservedRunningTime="2026-11-03 16:34:18.825003501 +0000 UTC m=+1420.404307374"
Nov 03 16:34:18.920622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:18.920585    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:19.451054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:19.451026    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:19.452496 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:19.452471    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:34:19.452728 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:19.452704    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:34:19.934591 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:19.934551    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 16:34:19.934769 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:19.934617    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 16:34:20.459268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:20.450634    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:34:20.459268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:20.450753    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:34:20.926248 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:20.921296    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 16:34:20.926248 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:20.921398    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 16:34:21.445021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:21.444498    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:34:21.445021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:21.444569    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:34:21.928273 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:21.923539    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 16:34:21.928273 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:21.923614    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 16:34:22.451311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:22.448670    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:34:22.451311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:22.448735    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:34:22.926174 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:22.926143    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 16:34:22.926933 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:22.926511    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 16:34:23.440360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:23.440327    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:34:23.440761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:23.440381    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:34:23.932439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:23.932402    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 16:34:23.932714 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:23.932681    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 16:34:24.440989 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:24.440721    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:34:24.442033 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:24.441999    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:34:24.921363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:24.921329    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 16:34:24.921549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:24.921384    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 16:34:25.185281 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.185173    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:25.185407 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.185348    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:25.440461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.440386    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 16:34:25.440461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.440440    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 16:34:25.898970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.898926    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh" event={"ID":"3fbe652f-e7ae-4938-bff9-c46e098caa35","Type":"ContainerStarted","Data":"55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4"}
Nov 03 16:34:25.900114 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.900089    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:25.921288 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.921256    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 03 16:34:25.921396 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:25.921318    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 03 16:34:26.124224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:26.121279    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh" podStartSLOduration=2.376939789 podStartE2EDuration="14.121259159s" totalImagesPullingTime="11.74431937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:26.065648561 +0000 UTC m=+1427.644952442" watchObservedRunningTime="2026-11-03 16:34:26.121259159 +0000 UTC m=+1427.700563016"
Nov 03 16:34:26.900221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:26.898058    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:26.913974 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:26.913945    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:26.922264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:26.917471    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:34:26.966925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:26.966891    5112 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-km6lh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:34:26.967249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:26.967175    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh" podUID="3fbe652f-e7ae-4938-bff9-c46e098caa35" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:34:27.378627 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:27.378541    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:27.381484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:27.381455    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:27.384313 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:27.384294    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:34:27.973162 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:27.973130    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:34:31.789428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.789394    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"]
Nov 03 16:34:31.794649 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.794625    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860130 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860089    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2nr\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-kube-api-access-vh2nr\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860137    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860407 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860319    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860407 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860366    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860407 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860382    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860566    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860691 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860669    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860707    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.860917 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.860898    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.899328 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.899290    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"]
Nov 03 16:34:31.962451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962413    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962519    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2nr\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-kube-api-access-vh2nr\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962554    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962664    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962708    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962778 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962731    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962809    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962876    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.962983 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.962902    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.963541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.963367    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istiod-ca-cert\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.963666 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.963631    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-data\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.964309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.963959    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-certs\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.964309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.964266    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.964484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.964380    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-credential-socket\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.966581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.966561    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-envoy\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:31.967302 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:31.967283    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-podinfo\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:32.020070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:32.020045    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-token\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:32.020290 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:32.020263    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2nr\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-kube-api-access-vh2nr\") pod \"e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") " pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:32.110262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:32.110172    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:32.562806 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:34:32.562773    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae2b292_a6a2_46e0_bcec_3027cdcd23d4.slice/crio-ebbe6d1e30daca56a71fbcaa8d7bbebbbccd35b39b7bf155ff068ea81a86d865 WatchSource:0}: Error finding container ebbe6d1e30daca56a71fbcaa8d7bbebbbccd35b39b7bf155ff068ea81a86d865: Status 404 returned error can't find the container with id ebbe6d1e30daca56a71fbcaa8d7bbebbbccd35b39b7bf155ff068ea81a86d865
Nov 03 16:34:32.563159 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:32.563134    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"]
Nov 03 16:34:32.566111 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:32.566077    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:32.566229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:32.566141    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:32.566229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:32.566169    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:34:33.025432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:33.025388    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" event={"ID":"cae2b292-a6a2-46e0-bcec-3027cdcd23d4","Type":"ContainerStarted","Data":"89485c187acd2d3e5c32ea43655c63920846b17502b518e2d9fba3aa269bcca0"}
Nov 03 16:34:33.025857 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:33.025465    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" event={"ID":"cae2b292-a6a2-46e0-bcec-3027cdcd23d4","Type":"ContainerStarted","Data":"ebbe6d1e30daca56a71fbcaa8d7bbebbbccd35b39b7bf155ff068ea81a86d865"}
Nov 03 16:34:33.114624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:33.114589    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:33.116182 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:33.116145    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 16:34:33.116387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:33.116260    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 16:34:33.227001 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:33.226944    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podStartSLOduration=2.226928748 podStartE2EDuration="2.226928748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:33.217415719 +0000 UTC m=+1434.796719573" watchObservedRunningTime="2026-11-03 16:34:33.226928748 +0000 UTC m=+1434.806232668"
Nov 03 16:34:34.110964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:34.110930    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 16:34:34.111378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:34.110986    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 16:34:35.111453 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:35.111419    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 16:34:35.111850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:35.111481    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 16:34:36.110547 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:36.110498    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 16:34:36.110730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:36.110574    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 16:34:37.111280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:37.111245    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 16:34:37.111699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:37.111297    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 16:34:37.924051 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:34:37.924017    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:34:38.110903 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:38.110863    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 16:34:38.111104 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:38.110923    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 16:34:39.111100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:39.111056    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 03 16:34:39.111648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:39.111157    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 03 16:34:40.116339 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:40.116311    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:40.116799 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:40.116771    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:34:40.117520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:34:40.117496    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:35:07.884686 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:35:07.884604    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:35:22.935732 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:22.935699    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"]
Nov 03 16:35:22.940804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:22.940785    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.041791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.041759    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-161f6056-serving-cert\""
Nov 03 16:35:23.049721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.049684    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"]
Nov 03 16:35:23.063781 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.063735    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66631cdb-da38-4c75-b437-a7fa7ea02438-openshift-service-ca\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.063964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.063798    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4mgz\" (UniqueName: \"kubernetes.io/projected/66631cdb-da38-4c75-b437-a7fa7ea02438-kube-api-access-j4mgz\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.063964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.063890    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-metrics-certs\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.063964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.063942    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.165839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.165790    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66631cdb-da38-4c75-b437-a7fa7ea02438-openshift-service-ca\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.165839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.165848    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4mgz\" (UniqueName: \"kubernetes.io/projected/66631cdb-da38-4c75-b437-a7fa7ea02438-kube-api-access-j4mgz\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.166075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.165909    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-metrics-certs\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.166075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.165955    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.166251 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:35:23.166192    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-serving-cert: secret "maas-api-e2e-derive-161f6056-serving-cert" not found
Nov 03 16:35:23.166344 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:35:23.166324    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls podName:66631cdb-da38-4c75-b437-a7fa7ea02438 nodeName:}" failed. No retries permitted until 2026-11-03 16:35:23.666300989 +0000 UTC m=+1485.245604825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls") pod "maas-api-e2e-derive-161f6056-6f8f675c7-75wq5" (UID: "66631cdb-da38-4c75-b437-a7fa7ea02438") : secret "maas-api-e2e-derive-161f6056-serving-cert" not found
Nov 03 16:35:23.166716 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.166692    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66631cdb-da38-4c75-b437-a7fa7ea02438-openshift-service-ca\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.168705 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.168679    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-metrics-certs\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.227323 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.227244    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4mgz\" (UniqueName: \"kubernetes.io/projected/66631cdb-da38-4c75-b437-a7fa7ea02438-kube-api-access-j4mgz\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.674803 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.674767    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.677393 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.677368    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls\") pod \"maas-api-e2e-derive-161f6056-6f8f675c7-75wq5\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:23.855886 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:23.855849    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:24.245593 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:35:24.245559    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66631cdb_da38_4c75_b437_a7fa7ea02438.slice/crio-3ef47ddca19ea47e5b93f5bfbe7cde9cd4ad52da8c6d2138ae5d9207f7e26389 WatchSource:0}: Error finding container 3ef47ddca19ea47e5b93f5bfbe7cde9cd4ad52da8c6d2138ae5d9207f7e26389: Status 404 returned error can't find the container with id 3ef47ddca19ea47e5b93f5bfbe7cde9cd4ad52da8c6d2138ae5d9207f7e26389
Nov 03 16:35:24.245919 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:24.245653    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"]
Nov 03 16:35:24.247098 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:24.247080    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:35:24.522377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:24.522291    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5" event={"ID":"66631cdb-da38-4c75-b437-a7fa7ea02438","Type":"ContainerStarted","Data":"3ef47ddca19ea47e5b93f5bfbe7cde9cd4ad52da8c6d2138ae5d9207f7e26389"}
Nov 03 16:35:25.531993 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:25.531955    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5" event={"ID":"66631cdb-da38-4c75-b437-a7fa7ea02438","Type":"ContainerStarted","Data":"dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31"}
Nov 03 16:35:25.532412 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:25.532047    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:25.657643 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:25.657580    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5" podStartSLOduration=2.8005872419999998 podStartE2EDuration="3.657563551s" totalImagesPullingTime="856.976309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:25.654174512 +0000 UTC m=+1487.233478384" watchObservedRunningTime="2026-11-03 16:35:25.657563551 +0000 UTC m=+1487.236867406"
Nov 03 16:35:31.546265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:31.546231    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:35:36.594970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:36.594934    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"]
Nov 03 16:35:37.886625 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:35:37.886579    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:35:38.766294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:38.766261    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:35:38.771806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:38.771774    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:35:38.781656 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:38.781623    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:35:38.786126 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:38.786099    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:35:41.595738 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:41.595671    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t" podUID="66613f0e-46a9-4637-abde-6edbb19354f2" containerName="payload-pre-processing" containerID="cri-o://14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c" gracePeriod=25
Nov 03 16:35:41.922120 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:41.922087    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:35:41.930626 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:41.930600    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66613f0e-46a9-4637-abde-6edbb19354f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66613f0e-46a9-4637-abde-6edbb19354f2-plugins-config-volume\") pod \"66613f0e-46a9-4637-abde-6edbb19354f2\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") "
Nov 03 16:35:41.930795 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:41.930766    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66613f0e-46a9-4637-abde-6edbb19354f2-kube-api-access-xf9rq\" (UniqueName: \"kubernetes.io/projected/66613f0e-46a9-4637-abde-6edbb19354f2-kube-api-access-xf9rq\") pod \"66613f0e-46a9-4637-abde-6edbb19354f2\" (UID: \"66613f0e-46a9-4637-abde-6edbb19354f2\") "
Nov 03 16:35:41.931044 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:41.931021    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66613f0e-46a9-4637-abde-6edbb19354f2-plugins-config-volume" pod "66613f0e-46a9-4637-abde-6edbb19354f2" (UID: "66613f0e-46a9-4637-abde-6edbb19354f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:35:41.931769 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:41.931709    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66613f0e-46a9-4637-abde-6edbb19354f2-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:35:41.933458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:41.933432    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66613f0e-46a9-4637-abde-6edbb19354f2-kube-api-access-xf9rq" pod "66613f0e-46a9-4637-abde-6edbb19354f2" (UID: "66613f0e-46a9-4637-abde-6edbb19354f2"). InnerVolumeSpecName "kube-api-access-xf9rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:35:42.034070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.034030    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9rq\" (UniqueName: \"kubernetes.io/projected/66613f0e-46a9-4637-abde-6edbb19354f2-kube-api-access-xf9rq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:35:42.698876 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.698840    5112 generic.go:356] "Generic (PLEG): container finished" podID="66613f0e-46a9-4637-abde-6edbb19354f2" containerID="14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c" exitCode=0
Nov 03 16:35:42.699386 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.698914    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"
Nov 03 16:35:42.699386 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.698924    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t" event={"ID":"66613f0e-46a9-4637-abde-6edbb19354f2","Type":"ContainerDied","Data":"14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c"}
Nov 03 16:35:42.699386 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.698976    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t" event={"ID":"66613f0e-46a9-4637-abde-6edbb19354f2","Type":"ContainerDied","Data":"5367e2e503ca9bc0a9b75586bdd0f29cbf830fc8fae352c1ec53210a9b990423"}
Nov 03 16:35:42.699386 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.699004    5112 scope.go:122] "RemoveContainer" containerID="14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c"
Nov 03 16:35:42.711089 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.711067    5112 scope.go:122] "RemoveContainer" containerID="14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c"
Nov 03 16:35:42.711371 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:35:42.711349    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c\": container with ID starting with 14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c not found: ID does not exist" containerID="14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c"
Nov 03 16:35:42.711457 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.711381    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c"} err="failed to get container status \"14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c\": rpc error: code = NotFound desc = could not find container \"14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c\": container with ID starting with 14ea0dedf44cd22bca769557c62f22915e92129d24310a0a0dca582aa1c78f8c not found: ID does not exist"
Nov 03 16:35:42.869350 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.869270    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"]
Nov 03 16:35:42.883266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:42.883234    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e998bdb4-7488c8d59-z9h8t"]
Nov 03 16:35:44.552731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:44.552695    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66613f0e-46a9-4637-abde-6edbb19354f2" path="/var/lib/kubelet/pods/66613f0e-46a9-4637-abde-6edbb19354f2/volumes"
Nov 03 16:35:53.254772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:53.254729    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"]
Nov 03 16:35:53.379540 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:53.379493    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"]
Nov 03 16:35:58.255234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.255156    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25" podUID="2346e75a-c070-4479-a39c-f074d22c38c5" containerName="payload-processing" containerID="cri-o://afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13" gracePeriod=25
Nov 03 16:35:58.395081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.394475    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2" podUID="a261207d-43f5-46e5-9b37-80da100d2abb" containerName="payload-pre-processing" containerID="cri-o://7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23" gracePeriod=25
Nov 03 16:35:58.593478 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.593447    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:35:58.686585 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.686146    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2346e75a-c070-4479-a39c-f074d22c38c5-kube-api-access-vsrw5\" (UniqueName: \"kubernetes.io/projected/2346e75a-c070-4479-a39c-f074d22c38c5-kube-api-access-vsrw5\") pod \"2346e75a-c070-4479-a39c-f074d22c38c5\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") "
Nov 03 16:35:58.686774 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.686737    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2346e75a-c070-4479-a39c-f074d22c38c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2346e75a-c070-4479-a39c-f074d22c38c5-plugins-config-volume\") pod \"2346e75a-c070-4479-a39c-f074d22c38c5\" (UID: \"2346e75a-c070-4479-a39c-f074d22c38c5\") "
Nov 03 16:35:58.687715 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.687678    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2346e75a-c070-4479-a39c-f074d22c38c5-plugins-config-volume" pod "2346e75a-c070-4479-a39c-f074d22c38c5" (UID: "2346e75a-c070-4479-a39c-f074d22c38c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:35:58.687862 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.687838    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2346e75a-c070-4479-a39c-f074d22c38c5-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:35:58.690026 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.689994    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2346e75a-c070-4479-a39c-f074d22c38c5-kube-api-access-vsrw5" pod "2346e75a-c070-4479-a39c-f074d22c38c5" (UID: "2346e75a-c070-4479-a39c-f074d22c38c5"). InnerVolumeSpecName "kube-api-access-vsrw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:35:58.722265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.722236    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:35:58.788615 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.788528    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a261207d-43f5-46e5-9b37-80da100d2abb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a261207d-43f5-46e5-9b37-80da100d2abb-plugins-config-volume\") pod \"a261207d-43f5-46e5-9b37-80da100d2abb\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") "
Nov 03 16:35:58.788850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.788832    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a261207d-43f5-46e5-9b37-80da100d2abb-kube-api-access-n6h2n\" (UniqueName: \"kubernetes.io/projected/a261207d-43f5-46e5-9b37-80da100d2abb-kube-api-access-n6h2n\") pod \"a261207d-43f5-46e5-9b37-80da100d2abb\" (UID: \"a261207d-43f5-46e5-9b37-80da100d2abb\") "
Nov 03 16:35:58.788898 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.788869    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a261207d-43f5-46e5-9b37-80da100d2abb-plugins-config-volume" pod "a261207d-43f5-46e5-9b37-80da100d2abb" (UID: "a261207d-43f5-46e5-9b37-80da100d2abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:35:58.789507 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.789482    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsrw5\" (UniqueName: \"kubernetes.io/projected/2346e75a-c070-4479-a39c-f074d22c38c5-kube-api-access-vsrw5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:35:58.789612 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.789509    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a261207d-43f5-46e5-9b37-80da100d2abb-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:35:58.791088 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.791058    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a261207d-43f5-46e5-9b37-80da100d2abb-kube-api-access-n6h2n" pod "a261207d-43f5-46e5-9b37-80da100d2abb" (UID: "a261207d-43f5-46e5-9b37-80da100d2abb"). InnerVolumeSpecName "kube-api-access-n6h2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:35:58.849179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.849147    5112 generic.go:356] "Generic (PLEG): container finished" podID="2346e75a-c070-4479-a39c-f074d22c38c5" containerID="afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13" exitCode=0
Nov 03 16:35:58.849383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.849193    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"
Nov 03 16:35:58.849383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.849237    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25" event={"ID":"2346e75a-c070-4479-a39c-f074d22c38c5","Type":"ContainerDied","Data":"afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13"}
Nov 03 16:35:58.849383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.849310    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25" event={"ID":"2346e75a-c070-4479-a39c-f074d22c38c5","Type":"ContainerDied","Data":"bb3520acba922669734e914a77357107e3f08e082edeba2b3716bb3fc4e0b303"}
Nov 03 16:35:58.849383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.849340    5112 scope.go:122] "RemoveContainer" containerID="afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13"
Nov 03 16:35:58.850892 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.850868    5112 generic.go:356] "Generic (PLEG): container finished" podID="a261207d-43f5-46e5-9b37-80da100d2abb" containerID="7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23" exitCode=0
Nov 03 16:35:58.851013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.850908    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"
Nov 03 16:35:58.851013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.850938    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2" event={"ID":"a261207d-43f5-46e5-9b37-80da100d2abb","Type":"ContainerDied","Data":"7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23"}
Nov 03 16:35:58.851013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.850978    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2" event={"ID":"a261207d-43f5-46e5-9b37-80da100d2abb","Type":"ContainerDied","Data":"a1253f1ee55414df6eaa01ac5a1fd77788a003812f94a6ae631d8a808939a7aa"}
Nov 03 16:35:58.862295 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.862279    5112 scope.go:122] "RemoveContainer" containerID="afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13"
Nov 03 16:35:58.862650 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:35:58.862623    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13\": container with ID starting with afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13 not found: ID does not exist" containerID="afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13"
Nov 03 16:35:58.862738 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.862656    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13"} err="failed to get container status \"afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13\": rpc error: code = NotFound desc = could not find container \"afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13\": container with ID starting with afcba62b4c26dd82918b8f5302d595a43fa4bdafd0abaa283022872552fa1d13 not found: ID does not exist"
Nov 03 16:35:58.862738 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.862684    5112 scope.go:122] "RemoveContainer" containerID="7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23"
Nov 03 16:35:58.875379 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.875349    5112 scope.go:122] "RemoveContainer" containerID="7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23"
Nov 03 16:35:58.875755 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:35:58.875683    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23\": container with ID starting with 7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23 not found: ID does not exist" containerID="7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23"
Nov 03 16:35:58.875755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.875718    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23"} err="failed to get container status \"7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23\": rpc error: code = NotFound desc = could not find container \"7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23\": container with ID starting with 7e637277d405bdc883e493596873aa146ba43bd47466b9556ac0906147108b23 not found: ID does not exist"
Nov 03 16:35:58.891103 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.891072    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6h2n\" (UniqueName: \"kubernetes.io/projected/a261207d-43f5-46e5-9b37-80da100d2abb-kube-api-access-n6h2n\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:35:58.989652 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:58.989612    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"]
Nov 03 16:35:59.003212 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:59.003167    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b774106-84d777c8d7-tlk25"]
Nov 03 16:35:59.116468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:59.116382    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"]
Nov 03 16:35:59.123874 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:35:59.123845    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b774106-7f4544c5f5-7w2x2"]
Nov 03 16:36:00.552573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:00.552543    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2346e75a-c070-4479-a39c-f074d22c38c5" path="/var/lib/kubelet/pods/2346e75a-c070-4479-a39c-f074d22c38c5/volumes"
Nov 03 16:36:00.553054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:00.553037    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a261207d-43f5-46e5-9b37-80da100d2abb" path="/var/lib/kubelet/pods/a261207d-43f5-46e5-9b37-80da100d2abb/volumes"
Nov 03 16:36:07.886844 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:36:07.886806    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:36:23.825536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:23.825457    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"]
Nov 03 16:36:23.887911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:23.887874    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"]
Nov 03 16:36:28.826325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:28.826282    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4" podUID="6da5d843-1527-45c5-b526-2f73c8512552" containerName="payload-processing" containerID="cri-o://851e67c1b912e9cb667a004b662b7e55a8427789e5f9eac2a21a941c0133f0c0" gracePeriod=25
Nov 03 16:36:28.888348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:28.888309    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9" podUID="444e6657-e43a-4738-aa04-1c78dee2db86" containerName="payload-pre-processing" containerID="cri-o://5e951d0d44ff990854207d1314875418676d57b166cf2dd2ee5eaa228db3514f" gracePeriod=25
Nov 03 16:36:29.138863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.138655    5112 generic.go:356] "Generic (PLEG): container finished" podID="444e6657-e43a-4738-aa04-1c78dee2db86" containerID="5e951d0d44ff990854207d1314875418676d57b166cf2dd2ee5eaa228db3514f" exitCode=0
Nov 03 16:36:29.138863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.138788    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9" event={"ID":"444e6657-e43a-4738-aa04-1c78dee2db86","Type":"ContainerDied","Data":"5e951d0d44ff990854207d1314875418676d57b166cf2dd2ee5eaa228db3514f"}
Nov 03 16:36:29.146762 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.146696    5112 generic.go:356] "Generic (PLEG): container finished" podID="6da5d843-1527-45c5-b526-2f73c8512552" containerID="851e67c1b912e9cb667a004b662b7e55a8427789e5f9eac2a21a941c0133f0c0" exitCode=0
Nov 03 16:36:29.147110 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.146932    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4" event={"ID":"6da5d843-1527-45c5-b526-2f73c8512552","Type":"ContainerDied","Data":"851e67c1b912e9cb667a004b662b7e55a8427789e5f9eac2a21a941c0133f0c0"}
Nov 03 16:36:29.577498 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.577148    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:36:29.647137 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.607099    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da5d843-1527-45c5-b526-2f73c8512552-kube-api-access-lgrps\" (UniqueName: \"kubernetes.io/projected/6da5d843-1527-45c5-b526-2f73c8512552-kube-api-access-lgrps\") pod \"6da5d843-1527-45c5-b526-2f73c8512552\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") "
Nov 03 16:36:29.647137 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.607363    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da5d843-1527-45c5-b526-2f73c8512552-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da5d843-1527-45c5-b526-2f73c8512552-plugins-config-volume\") pod \"6da5d843-1527-45c5-b526-2f73c8512552\" (UID: \"6da5d843-1527-45c5-b526-2f73c8512552\") "
Nov 03 16:36:29.689221 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.674576    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da5d843-1527-45c5-b526-2f73c8512552-kube-api-access-lgrps" pod "6da5d843-1527-45c5-b526-2f73c8512552" (UID: "6da5d843-1527-45c5-b526-2f73c8512552"). InnerVolumeSpecName "kube-api-access-lgrps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:36:29.697944 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.694690    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da5d843-1527-45c5-b526-2f73c8512552-plugins-config-volume" pod "6da5d843-1527-45c5-b526-2f73c8512552" (UID: "6da5d843-1527-45c5-b526-2f73c8512552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:36:29.777225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.766703    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgrps\" (UniqueName: \"kubernetes.io/projected/6da5d843-1527-45c5-b526-2f73c8512552-kube-api-access-lgrps\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:29.777225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.766747    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da5d843-1527-45c5-b526-2f73c8512552-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:29.852955 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:29.852923    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:36:30.076630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.076531    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444e6657-e43a-4738-aa04-1c78dee2db86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444e6657-e43a-4738-aa04-1c78dee2db86-plugins-config-volume\") pod \"444e6657-e43a-4738-aa04-1c78dee2db86\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") "
Nov 03 16:36:30.076792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.076676    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444e6657-e43a-4738-aa04-1c78dee2db86-kube-api-access-dkxr6\" (UniqueName: \"kubernetes.io/projected/444e6657-e43a-4738-aa04-1c78dee2db86-kube-api-access-dkxr6\") pod \"444e6657-e43a-4738-aa04-1c78dee2db86\" (UID: \"444e6657-e43a-4738-aa04-1c78dee2db86\") "
Nov 03 16:36:30.077002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.076972    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444e6657-e43a-4738-aa04-1c78dee2db86-plugins-config-volume" pod "444e6657-e43a-4738-aa04-1c78dee2db86" (UID: "444e6657-e43a-4738-aa04-1c78dee2db86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:36:30.077348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.077331    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/444e6657-e43a-4738-aa04-1c78dee2db86-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:30.078900 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.078856    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444e6657-e43a-4738-aa04-1c78dee2db86-kube-api-access-dkxr6" pod "444e6657-e43a-4738-aa04-1c78dee2db86" (UID: "444e6657-e43a-4738-aa04-1c78dee2db86"). InnerVolumeSpecName "kube-api-access-dkxr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:36:30.156504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.156467    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"
Nov 03 16:36:30.156504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.156481    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9" event={"ID":"444e6657-e43a-4738-aa04-1c78dee2db86","Type":"ContainerDied","Data":"22b59581db891674f296353dad65d9929071f781adcc42e9d712658e65224d7b"}
Nov 03 16:36:30.156745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.156553    5112 scope.go:122] "RemoveContainer" containerID="5e951d0d44ff990854207d1314875418676d57b166cf2dd2ee5eaa228db3514f"
Nov 03 16:36:30.170028 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.169989    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4" event={"ID":"6da5d843-1527-45c5-b526-2f73c8512552","Type":"ContainerDied","Data":"9a27576f2e970d7495a1ce63839f4a18fabe8d37713157891a820fad9e3a2862"}
Nov 03 16:36:30.170393 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.170358    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"
Nov 03 16:36:30.178401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.178376    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkxr6\" (UniqueName: \"kubernetes.io/projected/444e6657-e43a-4738-aa04-1c78dee2db86-kube-api-access-dkxr6\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:36:30.185989 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.185972    5112 scope.go:122] "RemoveContainer" containerID="851e67c1b912e9cb667a004b662b7e55a8427789e5f9eac2a21a941c0133f0c0"
Nov 03 16:36:30.341445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.341341    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"]
Nov 03 16:36:30.349986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.349957    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e6def865-dcf59658c-dhlh4"]
Nov 03 16:36:30.466573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.466542    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"]
Nov 03 16:36:30.481405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.481368    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e6def865-84c86bddff-kn4v9"]
Nov 03 16:36:30.554428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.554395    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444e6657-e43a-4738-aa04-1c78dee2db86" path="/var/lib/kubelet/pods/444e6657-e43a-4738-aa04-1c78dee2db86/volumes"
Nov 03 16:36:30.555044 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:36:30.555023    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da5d843-1527-45c5-b526-2f73c8512552" path="/var/lib/kubelet/pods/6da5d843-1527-45c5-b526-2f73c8512552/volumes"
Nov 03 16:36:37.883870 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:36:37.883831    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:37:07.890331 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:37:07.890286    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:37:08.907605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.907564    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"]
Nov 03 16:37:08.908568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908543    5112 cpu_manager.go:414] "containerMap: removing container" podUID="6da5d843-1527-45c5-b526-2f73c8512552" containerName="payload-processing"
Nov 03 16:37:08.908568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908566    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da5d843-1527-45c5-b526-2f73c8512552" containerName="payload-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908584    5112 cpu_manager.go:414] "containerMap: removing container" podUID="444e6657-e43a-4738-aa04-1c78dee2db86" containerName="payload-pre-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908591    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444e6657-e43a-4738-aa04-1c78dee2db86" containerName="payload-pre-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908620    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a261207d-43f5-46e5-9b37-80da100d2abb" containerName="payload-pre-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908627    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a261207d-43f5-46e5-9b37-80da100d2abb" containerName="payload-pre-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908660    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2346e75a-c070-4479-a39c-f074d22c38c5" containerName="payload-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908667    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2346e75a-c070-4479-a39c-f074d22c38c5" containerName="payload-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908691    5112 cpu_manager.go:414] "containerMap: removing container" podUID="66613f0e-46a9-4637-abde-6edbb19354f2" containerName="payload-pre-processing"
Nov 03 16:37:08.908754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908697    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66613f0e-46a9-4637-abde-6edbb19354f2" containerName="payload-pre-processing"
Nov 03 16:37:08.909125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908872    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2346e75a-c070-4479-a39c-f074d22c38c5" containerName="payload-processing"
Nov 03 16:37:08.909125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908891    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="66613f0e-46a9-4637-abde-6edbb19354f2" containerName="payload-pre-processing"
Nov 03 16:37:08.909125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908908    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da5d843-1527-45c5-b526-2f73c8512552" containerName="payload-processing"
Nov 03 16:37:08.909125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908921    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="444e6657-e43a-4738-aa04-1c78dee2db86" containerName="payload-pre-processing"
Nov 03 16:37:08.909125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.908932    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a261207d-43f5-46e5-9b37-80da100d2abb" containerName="payload-pre-processing"
Nov 03 16:37:08.912895 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.912874    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.991046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.991005    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.992519 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.992494    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.992794 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.992775    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.993128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.993107    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.993404 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.993382    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393e775e-2696-48b4-bc0b-5f5c34159a17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.993610 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.993591    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259kx\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-kube-api-access-259kx\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.993785 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.993768    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.994002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.993979    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:08.994120 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:08.994015    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.035617 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.035588    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"]
Nov 03 16:37:09.095540 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.095491    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.095735 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.095546    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.095735 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.095658    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.095856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.095736    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.095856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.095778    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.095967 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.095926    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.096023 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.095979    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393e775e-2696-48b4-bc0b-5f5c34159a17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.096078 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.096021    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259kx\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-kube-api-access-259kx\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.096078 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.096060    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.100280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.100254    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393e775e-2696-48b4-bc0b-5f5c34159a17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.101595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.101561    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.101769 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.101745    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.101846 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.101826    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.101981 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.101961    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.104252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.104177    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.104653 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.104630    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.156310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.156282    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-259kx\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-kube-api-access-259kx\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.156539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.156512    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-t2mvd\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.227793 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.227679    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:09.665974 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:37:09.665916    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393e775e_2696_48b4_bc0b_5f5c34159a17.slice/crio-98432e56b02229eb5f8f90f062f5d4944b2d82a2af251e94243ea25969a9dd67 WatchSource:0}: Error finding container 98432e56b02229eb5f8f90f062f5d4944b2d82a2af251e94243ea25969a9dd67: Status 404 returned error can't find the container with id 98432e56b02229eb5f8f90f062f5d4944b2d82a2af251e94243ea25969a9dd67
Nov 03 16:37:09.666827 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.666795    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"]
Nov 03 16:37:09.669308 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.669265    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:37:09.669405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.669357    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:37:09.669448 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:09.669403    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:37:10.604039 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:10.604002    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" event={"ID":"393e775e-2696-48b4-bc0b-5f5c34159a17","Type":"ContainerStarted","Data":"a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7"}
Nov 03 16:37:10.604522 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:10.604058    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" event={"ID":"393e775e-2696-48b4-bc0b-5f5c34159a17","Type":"ContainerStarted","Data":"98432e56b02229eb5f8f90f062f5d4944b2d82a2af251e94243ea25969a9dd67"}
Nov 03 16:37:11.228925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:11.228892    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:11.231397 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:11.231368    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 16:37:11.231555 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:11.231421    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 16:37:12.231688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:12.231631    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 16:37:12.232401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:12.232368    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 16:37:13.228192 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:13.228151    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 16:37:13.228400 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:13.228221    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 16:37:14.228179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:14.228137    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 16:37:14.228670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:14.228242    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 16:37:15.228887 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:15.228846    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 16:37:15.229341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:15.228917    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 16:37:16.228977 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:16.228940    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 16:37:16.229538 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:16.228998    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 16:37:17.229551 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:17.229516    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 16:37:17.229970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:17.229576    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 16:37:18.693347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:18.693310    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:18.694254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:18.694232    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:18.696069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:18.696043    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:37:18.811770 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:18.811701    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podStartSLOduration=10.811685087 podStartE2EDuration="10.811685087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:10.781057972 +0000 UTC m=+1592.360361829" watchObservedRunningTime="2026-11-03 16:37:18.811685087 +0000 UTC m=+1600.390989024"
Nov 03 16:37:26.822584 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:26.822546    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"]
Nov 03 16:37:26.823094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:26.822847    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5" podUID="66631cdb-da38-4c75-b437-a7fa7ea02438" containerName="maas-api" containerID="cri-o://dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31" gracePeriod=30
Nov 03 16:37:27.216041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.216012    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:37:27.310713 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.310686    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66631cdb-da38-4c75-b437-a7fa7ea02438-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66631cdb-da38-4c75-b437-a7fa7ea02438-openshift-service-ca\") pod \"66631cdb-da38-4c75-b437-a7fa7ea02438\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") "
Nov 03 16:37:27.310896 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.310742    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66631cdb-da38-4c75-b437-a7fa7ea02438-kube-api-access-j4mgz\" (UniqueName: \"kubernetes.io/projected/66631cdb-da38-4c75-b437-a7fa7ea02438-kube-api-access-j4mgz\") pod \"66631cdb-da38-4c75-b437-a7fa7ea02438\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") "
Nov 03 16:37:27.310966 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.310946    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-metrics-certs\") pod \"66631cdb-da38-4c75-b437-a7fa7ea02438\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") "
Nov 03 16:37:27.311065 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.311047    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls\") pod \"66631cdb-da38-4c75-b437-a7fa7ea02438\" (UID: \"66631cdb-da38-4c75-b437-a7fa7ea02438\") "
Nov 03 16:37:27.311188 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.311045    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66631cdb-da38-4c75-b437-a7fa7ea02438-openshift-service-ca" pod "66631cdb-da38-4c75-b437-a7fa7ea02438" (UID: "66631cdb-da38-4c75-b437-a7fa7ea02438"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:37:27.312181 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.312136    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66631cdb-da38-4c75-b437-a7fa7ea02438-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:27.313437 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.313403    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-metrics-certs" pod "66631cdb-da38-4c75-b437-a7fa7ea02438" (UID: "66631cdb-da38-4c75-b437-a7fa7ea02438"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:37:27.313541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.313463    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls" pod "66631cdb-da38-4c75-b437-a7fa7ea02438" (UID: "66631cdb-da38-4c75-b437-a7fa7ea02438"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:37:27.313541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.313503    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66631cdb-da38-4c75-b437-a7fa7ea02438-kube-api-access-j4mgz" pod "66631cdb-da38-4c75-b437-a7fa7ea02438" (UID: "66631cdb-da38-4c75-b437-a7fa7ea02438"). InnerVolumeSpecName "kube-api-access-j4mgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:37:27.413970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.413932    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4mgz\" (UniqueName: \"kubernetes.io/projected/66631cdb-da38-4c75-b437-a7fa7ea02438-kube-api-access-j4mgz\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:27.413970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.413972    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:27.414274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.413990    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66631cdb-da38-4c75-b437-a7fa7ea02438-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:27.745217 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.745160    5112 generic.go:356] "Generic (PLEG): container finished" podID="66631cdb-da38-4c75-b437-a7fa7ea02438" containerID="dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31" exitCode=0
Nov 03 16:37:27.745421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.745235    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"
Nov 03 16:37:27.745421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.745249    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5" event={"ID":"66631cdb-da38-4c75-b437-a7fa7ea02438","Type":"ContainerDied","Data":"dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31"}
Nov 03 16:37:27.745421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.745304    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5" event={"ID":"66631cdb-da38-4c75-b437-a7fa7ea02438","Type":"ContainerDied","Data":"3ef47ddca19ea47e5b93f5bfbe7cde9cd4ad52da8c6d2138ae5d9207f7e26389"}
Nov 03 16:37:27.745421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.745326    5112 scope.go:122] "RemoveContainer" containerID="dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31"
Nov 03 16:37:27.758095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.758076    5112 scope.go:122] "RemoveContainer" containerID="dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31"
Nov 03 16:37:27.758411 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:37:27.758387    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31\": container with ID starting with dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31 not found: ID does not exist" containerID="dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31"
Nov 03 16:37:27.758504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.758422    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31"} err="failed to get container status \"dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31\": rpc error: code = NotFound desc = could not find container \"dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31\": container with ID starting with dd8cc585f480b9c197711583b9670300bf43ccf7679c840de96e6523813eed31 not found: ID does not exist"
Nov 03 16:37:27.927051 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.927009    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"]
Nov 03 16:37:27.974508 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:27.974462    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-161f6056-6f8f675c7-75wq5"]
Nov 03 16:37:28.554183 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:28.554150    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66631cdb-da38-4c75-b437-a7fa7ea02438" path="/var/lib/kubelet/pods/66631cdb-da38-4c75-b437-a7fa7ea02438/volumes"
Nov 03 16:37:37.887703 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:37:37.886850    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:37:52.253543 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:52.253461    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"]
Nov 03 16:37:52.360252 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:52.360217    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"]
Nov 03 16:37:53.911346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:53.911303    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"]
Nov 03 16:37:53.912658 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:53.912632    5112 cpu_manager.go:414] "containerMap: removing container" podUID="66631cdb-da38-4c75-b437-a7fa7ea02438" containerName="maas-api"
Nov 03 16:37:53.913032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:53.913012    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66631cdb-da38-4c75-b437-a7fa7ea02438" containerName="maas-api"
Nov 03 16:37:53.913492 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:53.913473    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="66631cdb-da38-4c75-b437-a7fa7ea02438" containerName="maas-api"
Nov 03 16:37:53.920021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:53.920002    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.029032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.028996    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"]
Nov 03 16:37:54.038028 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.037992    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038491 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038459    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038579 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038510    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038579 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038547    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038686 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038606    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzsts\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-kube-api-access-wzsts\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038686 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038632    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038794 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038699    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038848 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038816    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.038905 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.038872    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.140646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.140615    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.140850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.140689    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzsts\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-kube-api-access-wzsts\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.140850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.140718    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.140850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.140783    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.140893    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.140953    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.141039    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141323 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.141297    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.141488    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.141498    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.141557    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.141311    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.141788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.141718    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.142685 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.142660    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.143356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.143330    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.145657 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.145612    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.197888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.197817    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzsts\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-kube-api-access-wzsts\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.198000 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.197932    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-77r8r\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.240810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.240781    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:54.732583 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:37:54.732541    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2815b0ea_11fe_42fd_9b0f_e4f438ef3873.slice/crio-4c6b7d2d83d29ba7491b8e36d84139f94abbe7264cf9bb110d289e42c6a54d4b WatchSource:0}: Error finding container 4c6b7d2d83d29ba7491b8e36d84139f94abbe7264cf9bb110d289e42c6a54d4b: Status 404 returned error can't find the container with id 4c6b7d2d83d29ba7491b8e36d84139f94abbe7264cf9bb110d289e42c6a54d4b
Nov 03 16:37:54.733314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.733286    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"]
Nov 03 16:37:54.735176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.735139    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:37:54.735312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.735229    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:37:54.735312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:54.735264    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 16:37:55.083278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:55.082707    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" event={"ID":"2815b0ea-11fe-42fd-9b0f-e4f438ef3873","Type":"ContainerStarted","Data":"88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a"}
Nov 03 16:37:55.083278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:55.082804    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" event={"ID":"2815b0ea-11fe-42fd-9b0f-e4f438ef3873","Type":"ContainerStarted","Data":"4c6b7d2d83d29ba7491b8e36d84139f94abbe7264cf9bb110d289e42c6a54d4b"}
Nov 03 16:37:55.241159 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:55.241127    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:37:55.242781 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:55.242748    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:37:55.242935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:55.242808    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:37:56.241481 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:56.241446    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:37:56.241917 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:56.241514    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:37:57.241229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.241184    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:37:57.241402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.241246    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:37:57.276419 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.255598    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98" podUID="aa5978ac-f897-4bfd-b3fd-60a9847f633b" containerName="payload-processing" containerID="cri-o://44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac" gracePeriod=25
Nov 03 16:37:57.372227 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.370468    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh" podUID="212091e5-3ae7-4d1a-9510-5d0a0b654f6f" containerName="payload-pre-processing" containerID="cri-o://bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6" gracePeriod=25
Nov 03 16:37:57.746306 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.746240    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:37:57.836211 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.836171    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:37:57.844980 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.844394    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5978ac-f897-4bfd-b3fd-60a9847f633b-kube-api-access-sr9pf\" (UniqueName: \"kubernetes.io/projected/aa5978ac-f897-4bfd-b3fd-60a9847f633b-kube-api-access-sr9pf\") pod \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") "
Nov 03 16:37:57.844980 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.844665    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5978ac-f897-4bfd-b3fd-60a9847f633b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5978ac-f897-4bfd-b3fd-60a9847f633b-plugins-config-volume\") pod \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\" (UID: \"aa5978ac-f897-4bfd-b3fd-60a9847f633b\") "
Nov 03 16:37:57.844980 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.844946    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5978ac-f897-4bfd-b3fd-60a9847f633b-plugins-config-volume" pod "aa5978ac-f897-4bfd-b3fd-60a9847f633b" (UID: "aa5978ac-f897-4bfd-b3fd-60a9847f633b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:37:57.845344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.845328    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5978ac-f897-4bfd-b3fd-60a9847f633b-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:57.846480 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.846458    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5978ac-f897-4bfd-b3fd-60a9847f633b-kube-api-access-sr9pf" pod "aa5978ac-f897-4bfd-b3fd-60a9847f633b" (UID: "aa5978ac-f897-4bfd-b3fd-60a9847f633b"). InnerVolumeSpecName "kube-api-access-sr9pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:37:57.889142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.889063    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podStartSLOduration=4.889042688 podStartE2EDuration="4.889042688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:55.202015651 +0000 UTC m=+1636.781319528" watchObservedRunningTime="2026-11-03 16:37:57.889042688 +0000 UTC m=+1639.468346544"
Nov 03 16:37:57.946619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.946589    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-plugins-config-volume\") pod \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") "
Nov 03 16:37:57.946814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.946723    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-kube-api-access-wrzx5\" (UniqueName: \"kubernetes.io/projected/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-kube-api-access-wrzx5\") pod \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\" (UID: \"212091e5-3ae7-4d1a-9510-5d0a0b654f6f\") "
Nov 03 16:37:57.947014 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.946990    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-plugins-config-volume" pod "212091e5-3ae7-4d1a-9510-5d0a0b654f6f" (UID: "212091e5-3ae7-4d1a-9510-5d0a0b654f6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:37:57.947562 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.947539    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9pf\" (UniqueName: \"kubernetes.io/projected/aa5978ac-f897-4bfd-b3fd-60a9847f633b-kube-api-access-sr9pf\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:57.947562 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.947561    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:57.949347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:57.949325    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-kube-api-access-wrzx5" pod "212091e5-3ae7-4d1a-9510-5d0a0b654f6f" (UID: "212091e5-3ae7-4d1a-9510-5d0a0b654f6f"). InnerVolumeSpecName "kube-api-access-wrzx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:37:58.049343 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.049308    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrzx5\" (UniqueName: \"kubernetes.io/projected/212091e5-3ae7-4d1a-9510-5d0a0b654f6f-kube-api-access-wrzx5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:37:58.146724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.146631    5112 generic.go:356] "Generic (PLEG): container finished" podID="aa5978ac-f897-4bfd-b3fd-60a9847f633b" containerID="44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac" exitCode=0
Nov 03 16:37:58.146724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.146695    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"
Nov 03 16:37:58.146724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.146715    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98" event={"ID":"aa5978ac-f897-4bfd-b3fd-60a9847f633b","Type":"ContainerDied","Data":"44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac"}
Nov 03 16:37:58.146960 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.146801    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98" event={"ID":"aa5978ac-f897-4bfd-b3fd-60a9847f633b","Type":"ContainerDied","Data":"edff6266047874485cc34cd2ce93cf2a671faf88ddc21da25765283bd7492bd4"}
Nov 03 16:37:58.146960 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.146837    5112 scope.go:122] "RemoveContainer" containerID="44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac"
Nov 03 16:37:58.148878 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.148857    5112 generic.go:356] "Generic (PLEG): container finished" podID="212091e5-3ae7-4d1a-9510-5d0a0b654f6f" containerID="bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6" exitCode=0
Nov 03 16:37:58.149087 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.148924    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh" event={"ID":"212091e5-3ae7-4d1a-9510-5d0a0b654f6f","Type":"ContainerDied","Data":"bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6"}
Nov 03 16:37:58.149263 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.149243    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh" event={"ID":"212091e5-3ae7-4d1a-9510-5d0a0b654f6f","Type":"ContainerDied","Data":"3d0d1cc9da5b79d1c496afabb19f725bc0eef385ae99da0dee08e64dd6a8d5c9"}
Nov 03 16:37:58.149263 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.148946    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"
Nov 03 16:37:58.187708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.187613    5112 scope.go:122] "RemoveContainer" containerID="44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac"
Nov 03 16:37:58.188894 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:37:58.188496    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac\": container with ID starting with 44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac not found: ID does not exist" containerID="44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac"
Nov 03 16:37:58.188894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.188533    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac"} err="failed to get container status \"44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac\": rpc error: code = NotFound desc = could not find container \"44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac\": container with ID starting with 44613c86903aad70345911017e904e86cd1d44ab1341b446155c863c130ddeac not found: ID does not exist"
Nov 03 16:37:58.188894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.188559    5112 scope.go:122] "RemoveContainer" containerID="bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6"
Nov 03 16:37:58.204720 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.204706    5112 scope.go:122] "RemoveContainer" containerID="bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6"
Nov 03 16:37:58.205030 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:37:58.204982    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6\": container with ID starting with bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6 not found: ID does not exist" containerID="bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6"
Nov 03 16:37:58.205112 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.205042    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6"} err="failed to get container status \"bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6\": rpc error: code = NotFound desc = could not find container \"bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6\": container with ID starting with bc2450b73f00b00a731e37d1549c7304f0f15b724832d92ba7d9d6cc231390d6 not found: ID does not exist"
Nov 03 16:37:58.241381 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.241354    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:37:58.241480 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.241404    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:37:58.316301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.316135    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"]
Nov 03 16:37:58.324887 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.324857    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06f127ec-77699d88c5-zwr98"]
Nov 03 16:37:58.445537 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.445448    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"]
Nov 03 16:37:58.450745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.450718    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06f127ec-c5cc876c-m78nh"]
Nov 03 16:37:58.552026 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.551998    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212091e5-3ae7-4d1a-9510-5d0a0b654f6f" path="/var/lib/kubelet/pods/212091e5-3ae7-4d1a-9510-5d0a0b654f6f/volumes"
Nov 03 16:37:58.552492 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:58.552476    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5978ac-f897-4bfd-b3fd-60a9847f633b" path="/var/lib/kubelet/pods/aa5978ac-f897-4bfd-b3fd-60a9847f633b/volumes"
Nov 03 16:37:59.241918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:59.241883    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:37:59.242089 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:37:59.241952    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:38:00.241944 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:00.241909    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:38:00.242349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:00.241958    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:38:01.242369 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:01.242327    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:38:01.242881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:01.242390    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:38:02.242219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:02.241403    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 16:38:02.242219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:02.241572    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 16:38:03.753887 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:03.753850    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:38:03.754362 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:03.754238    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:38:03.755684 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:03.755598    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:38:07.886384 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:38:07.886332    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:38:36.642187 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.642142    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"]
Nov 03 16:38:36.643267 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.643243    5112 cpu_manager.go:414] "containerMap: removing container" podUID="aa5978ac-f897-4bfd-b3fd-60a9847f633b" containerName="payload-processing"
Nov 03 16:38:36.643267 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.643264    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5978ac-f897-4bfd-b3fd-60a9847f633b" containerName="payload-processing"
Nov 03 16:38:36.643426 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.643296    5112 cpu_manager.go:414] "containerMap: removing container" podUID="212091e5-3ae7-4d1a-9510-5d0a0b654f6f" containerName="payload-pre-processing"
Nov 03 16:38:36.643426 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.643303    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212091e5-3ae7-4d1a-9510-5d0a0b654f6f" containerName="payload-pre-processing"
Nov 03 16:38:36.643499 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.643453    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5978ac-f897-4bfd-b3fd-60a9847f633b" containerName="payload-processing"
Nov 03 16:38:36.643499 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.643466    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="212091e5-3ae7-4d1a-9510-5d0a0b654f6f" containerName="payload-pre-processing"
Nov 03 16:38:36.648483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.648456    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.706857 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.706813    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"]
Nov 03 16:38:36.747157 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.747122    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9635db0d-serving-cert\""
Nov 03 16:38:36.752741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.752711    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-metrics-certs\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.753052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.752884    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.753052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.752919    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6ee17-7736-44a8-8a55-9ad954129b8b-openshift-service-ca\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.753052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.752958    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wf5p\" (UniqueName: \"kubernetes.io/projected/07a6ee17-7736-44a8-8a55-9ad954129b8b-kube-api-access-2wf5p\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.854491 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.854454    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.854656 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.854501    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6ee17-7736-44a8-8a55-9ad954129b8b-openshift-service-ca\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.854656 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.854553    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wf5p\" (UniqueName: \"kubernetes.io/projected/07a6ee17-7736-44a8-8a55-9ad954129b8b-kube-api-access-2wf5p\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.854656 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:38:36.854634    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-serving-cert: secret "maas-api-e2e-mt-9635db0d-serving-cert" not found
Nov 03 16:38:36.854830 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:38:36.854726    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls podName:07a6ee17-7736-44a8-8a55-9ad954129b8b nodeName:}" failed. No retries permitted until 2026-11-03 16:38:37.354704603 +0000 UTC m=+1678.934008443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls") pod "maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9" (UID: "07a6ee17-7736-44a8-8a55-9ad954129b8b") : secret "maas-api-e2e-mt-9635db0d-serving-cert" not found
Nov 03 16:38:36.854830 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.854765    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-metrics-certs\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.855311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.855289    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6ee17-7736-44a8-8a55-9ad954129b8b-openshift-service-ca\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.857616 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.857597    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-metrics-certs\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:36.913020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:36.912985    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wf5p\" (UniqueName: \"kubernetes.io/projected/07a6ee17-7736-44a8-8a55-9ad954129b8b-kube-api-access-2wf5p\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:37.365927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:37.365832    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:37.368598 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:37.368573    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls\") pod \"maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:37.566298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:37.566263    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:37.885084 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:38:37.885004    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:38:37.953345 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:37.953314    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"]
Nov 03 16:38:37.955113 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:38:37.955085    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a6ee17_7736_44a8_8a55_9ad954129b8b.slice/crio-cc4eb98ccae86cef76c31c479e460f0e07e01b9daebc18f811c729b701dd44ef WatchSource:0}: Error finding container cc4eb98ccae86cef76c31c479e460f0e07e01b9daebc18f811c729b701dd44ef: Status 404 returned error can't find the container with id cc4eb98ccae86cef76c31c479e460f0e07e01b9daebc18f811c729b701dd44ef
Nov 03 16:38:38.611587 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:38.611544    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9" event={"ID":"07a6ee17-7736-44a8-8a55-9ad954129b8b","Type":"ContainerStarted","Data":"cc4eb98ccae86cef76c31c479e460f0e07e01b9daebc18f811c729b701dd44ef"}
Nov 03 16:38:39.625663 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:39.625569    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9" event={"ID":"07a6ee17-7736-44a8-8a55-9ad954129b8b","Type":"ContainerStarted","Data":"4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342"}
Nov 03 16:38:39.626810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:39.626789    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:39.743971 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:39.743917    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9" podStartSLOduration=2.908550461 podStartE2EDuration="3.743903848s" totalImagesPullingTime="835.353387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:39.740256996 +0000 UTC m=+1681.319560868" watchObservedRunningTime="2026-11-03 16:38:39.743903848 +0000 UTC m=+1681.323207703"
Nov 03 16:38:46.652705 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:46.652675    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:38:56.668827 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.668786    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"]
Nov 03 16:38:56.675838 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.675816    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.773489 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.773458    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-509f14-serving-cert\""
Nov 03 16:38:56.781132 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.781102    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"]
Nov 03 16:38:56.788989 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.788885    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pp5\" (UniqueName: \"kubernetes.io/projected/2478e2ce-c49e-483d-9ce4-60f0283a323e-kube-api-access-g7pp5\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.788989 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.788949    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.789315 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.789128    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2478e2ce-c49e-483d-9ce4-60f0283a323e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.789447 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.789424    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.891919 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.891867    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pp5\" (UniqueName: \"kubernetes.io/projected/2478e2ce-c49e-483d-9ce4-60f0283a323e-kube-api-access-g7pp5\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.892124 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.891939    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.892124 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.892000    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2478e2ce-c49e-483d-9ce4-60f0283a323e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.892299 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.892147    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.892583 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:38:56.892508    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-serving-cert: secret "maas-api-e2e-shared-b-w2-509f14-serving-cert" not found
Nov 03 16:38:56.892654 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:38:56.892600    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls podName:2478e2ce-c49e-483d-9ce4-60f0283a323e nodeName:}" failed. No retries permitted until 2026-11-03 16:38:57.392575422 +0000 UTC m=+1698.971879272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz" (UID: "2478e2ce-c49e-483d-9ce4-60f0283a323e") : secret "maas-api-e2e-shared-b-w2-509f14-serving-cert" not found
Nov 03 16:38:56.893006 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.892984    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2478e2ce-c49e-483d-9ce4-60f0283a323e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.894670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.894644    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:56.980271 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:56.969956    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pp5\" (UniqueName: \"kubernetes.io/projected/2478e2ce-c49e-483d-9ce4-60f0283a323e-kube-api-access-g7pp5\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:57.405169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:57.405128    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:57.407642 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:57.407614    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:57.592088 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:57.592051    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:58.063153 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:58.063109    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"]
Nov 03 16:38:58.063973 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:38:58.063944    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2478e2ce_c49e_483d_9ce4_60f0283a323e.slice/crio-a8119960024f9f3f13571b0b3dd81c94af5ccac9acc511f73096410bd184b5f7 WatchSource:0}: Error finding container a8119960024f9f3f13571b0b3dd81c94af5ccac9acc511f73096410bd184b5f7: Status 404 returned error can't find the container with id a8119960024f9f3f13571b0b3dd81c94af5ccac9acc511f73096410bd184b5f7
Nov 03 16:38:59.002076 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:59.002042    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz" event={"ID":"2478e2ce-c49e-483d-9ce4-60f0283a323e","Type":"ContainerStarted","Data":"9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411"}
Nov 03 16:38:59.002254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:59.002103    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz" event={"ID":"2478e2ce-c49e-483d-9ce4-60f0283a323e","Type":"ContainerStarted","Data":"a8119960024f9f3f13571b0b3dd81c94af5ccac9acc511f73096410bd184b5f7"}
Nov 03 16:38:59.002254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:59.002156    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:38:59.133033 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:38:59.132978    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz" podStartSLOduration=2.321511743 podStartE2EDuration="3.132965638s" totalImagesPullingTime="811.453895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:59.131141965 +0000 UTC m=+1700.710445843" watchObservedRunningTime="2026-11-03 16:38:59.132965638 +0000 UTC m=+1700.712269493"
Nov 03 16:39:05.017093 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:05.017066    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 16:39:07.884692 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:39:07.884650    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:39:37.884262 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:39:37.884229    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:39:48.442432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:48.442385    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"]
Nov 03 16:39:48.442941 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:48.442647    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" containerID="cri-o://af29c24524df726c4ccf3a783c82bdb37a3119f988b28216dbb70e29aa44f15e" gracePeriod=30
Nov 03 16:39:48.987257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:48.987189    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"]
Nov 03 16:39:48.987796 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:48.987761    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" containerID="cri-o://ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07" gracePeriod=30
Nov 03 16:39:53.019791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.019750    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:39:53.020186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.019829    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:39:53.851428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.851397    5112 generic.go:356] "Generic (PLEG): container finished" podID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerID="af29c24524df726c4ccf3a783c82bdb37a3119f988b28216dbb70e29aa44f15e" exitCode=0
Nov 03 16:39:53.851607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.851472    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" event={"ID":"40eb21d3-2ad2-4c2c-a22c-7867a2352537","Type":"ContainerDied","Data":"af29c24524df726c4ccf3a783c82bdb37a3119f988b28216dbb70e29aa44f15e"}
Nov 03 16:39:53.851607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.851533    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx" event={"ID":"40eb21d3-2ad2-4c2c-a22c-7867a2352537","Type":"ContainerDied","Data":"909d87a0abd0fdef6d071fde140ab21facab5f22acea36a46bc951330b601689"}
Nov 03 16:39:53.851607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.851555    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909d87a0abd0fdef6d071fde140ab21facab5f22acea36a46bc951330b601689"
Nov 03 16:39:53.859722 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.859690    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:39:53.859963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.859935    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:39:53.867968 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.867944    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.954822    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-podinfo\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.954893    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-certs\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.954934    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-credential-socket\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.955066    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-socket\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.955103    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-kube-api-access-f24vk\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-kube-api-access-f24vk\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.955163    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istiod-ca-cert\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.955286    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-envoy\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.955488    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-token\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-token\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.955646    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-data\") pod \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\" (UID: \"40eb21d3-2ad2-4c2c-a22c-7867a2352537\") "
Nov 03 16:39:53.965226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.963978    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-data" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:53.972230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.970721    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-credential-socket" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:53.972230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.970920    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-socket" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:53.972484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.972293    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-certs" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:53.974142 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.973503    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istiod-ca-cert" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:39:53.979270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.979082    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-kube-api-access-f24vk" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "kube-api-access-f24vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:53.988786 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.988414    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-podinfo" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:39:53.989523 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.989478    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-token" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:53.998727 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:53.990360    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-envoy" pod "40eb21d3-2ad2-4c2c-a22c-7867a2352537" (UID: "40eb21d3-2ad2-4c2c-a22c-7867a2352537"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:54.062119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062068    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062110    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062125    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062140    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062153    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f24vk\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-kube-api-access-f24vk\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062169    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062183    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062211    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.062772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.062225    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eb21d3-2ad2-4c2c-a22c-7867a2352537-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.319403 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.319355    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:39:54.368764 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.368727    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441034a0-b050-4421-8294-e6401e9049c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441034a0-b050-4421-8294-e6401e9049c4-istiod-ca-cert\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.368932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.368888    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/441034a0-b050-4421-8294-e6401e9049c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441034a0-b050-4421-8294-e6401e9049c4-istio-podinfo\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369020    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-istio-token\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-istio-token\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369092    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-credential-socket\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369170 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369144    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441034a0-b050-4421-8294-e6401e9049c4-istiod-ca-cert" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:39:54.369312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369291    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-kube-api-access-d9pb7\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-kube-api-access-d9pb7\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369398 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369374    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-certs\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369452 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369423    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-envoy\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369510 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369492    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-data\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369608 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369570    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-socket\") pod \"441034a0-b050-4421-8294-e6401e9049c4\" (UID: \"441034a0-b050-4421-8294-e6401e9049c4\") "
Nov 03 16:39:54.369791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.369767    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-credential-socket" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:54.370600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.370145    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-socket" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:54.370600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.370521    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.370600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.370543    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.370600 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.370558    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441034a0-b050-4421-8294-e6401e9049c4-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.371257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.371172    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-data" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:54.371770 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.371726    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/441034a0-b050-4421-8294-e6401e9049c4-istio-podinfo" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:39:54.372169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.372145    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-certs" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:54.372457 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.372432    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-istio-token" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:54.372895 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.372873    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-envoy" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:39:54.373055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.373030    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-kube-api-access-d9pb7" pod "441034a0-b050-4421-8294-e6401e9049c4" (UID: "441034a0-b050-4421-8294-e6401e9049c4"). InnerVolumeSpecName "kube-api-access-d9pb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:39:54.473134 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.473052    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441034a0-b050-4421-8294-e6401e9049c4-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.473134 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.473091    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.473134 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.473106    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pb7\" (UniqueName: \"kubernetes.io/projected/441034a0-b050-4421-8294-e6401e9049c4-kube-api-access-d9pb7\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.473134 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.473123    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.473480 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.473138    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.473480 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.473152    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441034a0-b050-4421-8294-e6401e9049c4-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:39:54.873802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.873707    5112 generic.go:356] "Generic (PLEG): container finished" podID="441034a0-b050-4421-8294-e6401e9049c4" containerID="ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07" exitCode=0
Nov 03 16:39:54.873802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.873765    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"
Nov 03 16:39:54.874018 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.873795    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" event={"ID":"441034a0-b050-4421-8294-e6401e9049c4","Type":"ContainerDied","Data":"ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07"}
Nov 03 16:39:54.874018 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.873859    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94" event={"ID":"441034a0-b050-4421-8294-e6401e9049c4","Type":"ContainerDied","Data":"a71a12b7f689ca43e4fd628435b5789833a9295bc8ef5c86cc47f04d70d6df30"}
Nov 03 16:39:54.874018 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.873890    5112 scope.go:122] "RemoveContainer" containerID="ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07"
Nov 03 16:39:54.874151 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.874129    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"
Nov 03 16:39:54.886898 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.886869    5112 scope.go:122] "RemoveContainer" containerID="ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07"
Nov 03 16:39:54.887228 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:39:54.887191    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07\": container with ID starting with ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07 not found: ID does not exist" containerID="ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07"
Nov 03 16:39:54.887318 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:54.887232    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07"} err="failed to get container status \"ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07\": rpc error: code = NotFound desc = could not find container \"ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07\": container with ID starting with ccba74eb9e4cb1de2c712a09dcfebe2d5ce272eba617e23ef141cc6268569f07 not found: ID does not exist"
Nov 03 16:39:55.040760 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:55.040725    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"]
Nov 03 16:39:55.089918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:55.089884    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-tkb94"]
Nov 03 16:39:55.165309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:55.165270    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"]
Nov 03 16:39:55.174507 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:55.174474    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-nfrxx"]
Nov 03 16:39:56.552074 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:56.552040    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" path="/var/lib/kubelet/pods/40eb21d3-2ad2-4c2c-a22c-7867a2352537/volumes"
Nov 03 16:39:56.552610 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:56.552592    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441034a0-b050-4421-8294-e6401e9049c4" path="/var/lib/kubelet/pods/441034a0-b050-4421-8294-e6401e9049c4/volumes"
Nov 03 16:39:58.279233 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:58.279175    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"]
Nov 03 16:39:58.279675 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:39:58.279433    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy" containerID="cri-o://13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2" gracePeriod=30
Nov 03 16:40:03.586124 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.586087    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:40:03.656541 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656497    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487f67a9-7a18-4994-800c-66c7690aff43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f67a9-7a18-4994-800c-66c7690aff43-istiod-ca-cert\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.656761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656555    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/487f67a9-7a18-4994-800c-66c7690aff43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f67a9-7a18-4994-800c-66c7690aff43-istio-podinfo\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.656761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656592    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-envoy\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.656761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656675    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-credential-socket\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.656761 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656704    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-socket\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.657013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656798    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-istio-token\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-istio-token\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.657013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656833    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-certs\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.657013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656872    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-kube-api-access-cjxvx\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-kube-api-access-cjxvx\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.657013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.656987    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487f67a9-7a18-4994-800c-66c7690aff43-istiod-ca-cert" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:40:03.657500 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.657360    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-credential-socket" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:03.659114 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.657709    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f67a9-7a18-4994-800c-66c7690aff43-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.659114 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.657736    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.659114 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.658456    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-socket" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:03.659114 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.658742    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-certs" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:03.667866 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.666624    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/487f67a9-7a18-4994-800c-66c7690aff43-istio-podinfo" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:40:03.674817 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.674789    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-kube-api-access-cjxvx" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "kube-api-access-cjxvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:40:03.680798 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.680768    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-envoy" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:03.681099 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.681072    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-istio-token" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:40:03.758776 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.758744    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-data\") pod \"487f67a9-7a18-4994-800c-66c7690aff43\" (UID: \"487f67a9-7a18-4994-800c-66c7690aff43\") "
Nov 03 16:40:03.759088 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759061    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-data" pod "487f67a9-7a18-4994-800c-66c7690aff43" (UID: "487f67a9-7a18-4994-800c-66c7690aff43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:40:03.759517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759495    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.759574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759531    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.759574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759547    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjxvx\" (UniqueName: \"kubernetes.io/projected/487f67a9-7a18-4994-800c-66c7690aff43-kube-api-access-cjxvx\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.759574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759560    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.759665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759574    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f67a9-7a18-4994-800c-66c7690aff43-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.759665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759585    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.759665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.759598    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f67a9-7a18-4994-800c-66c7690aff43-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:40:03.953863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.953826    5112 generic.go:356] "Generic (PLEG): container finished" podID="487f67a9-7a18-4994-800c-66c7690aff43" containerID="13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2" exitCode=0
Nov 03 16:40:03.954066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.953880    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"
Nov 03 16:40:03.954066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.953911    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" event={"ID":"487f67a9-7a18-4994-800c-66c7690aff43","Type":"ContainerDied","Data":"13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2"}
Nov 03 16:40:03.954066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.953999    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df" event={"ID":"487f67a9-7a18-4994-800c-66c7690aff43","Type":"ContainerDied","Data":"32b3b1884f9684927206d014bd3e9801ad9ccc0270b893f181dbcfa1266f52ab"}
Nov 03 16:40:03.954066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.954023    5112 scope.go:122] "RemoveContainer" containerID="13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2"
Nov 03 16:40:03.968525 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.968505    5112 scope.go:122] "RemoveContainer" containerID="13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2"
Nov 03 16:40:03.968901 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:40:03.968870    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2\": container with ID starting with 13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2 not found: ID does not exist" containerID="13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2"
Nov 03 16:40:03.968966 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:03.968912    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2"} err="failed to get container status \"13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2\": rpc error: code = NotFound desc = could not find container \"13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2\": container with ID starting with 13425e8c6c235994cd8f0b19b3f0fd30253b797e4bd46ac759c608ac001ca9a2 not found: ID does not exist"
Nov 03 16:40:04.118787 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:04.118756    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"]
Nov 03 16:40:04.164939 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:04.164908    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-6j6df"]
Nov 03 16:40:04.551814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:04.551774    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487f67a9-7a18-4994-800c-66c7690aff43" path="/var/lib/kubelet/pods/487f67a9-7a18-4994-800c-66c7690aff43/volumes"
Nov 03 16:40:07.885495 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:40:07.885448    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:40:37.887778 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:40:37.887740    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:40:38.826353 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:38.826321    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:40:38.842826 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:38.842795    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:40:38.847146 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:38.847119    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:40:38.862930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:38.862895    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:40:39.298573 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:40:39.298538    5112 scope.go:122] "RemoveContainer" containerID="af29c24524df726c4ccf3a783c82bdb37a3119f988b28216dbb70e29aa44f15e"
Nov 03 16:41:07.888867 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:41:07.888771    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:41:37.890988 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:41:37.890945    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:42:07.887696 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:42:07.887655    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:42:37.886649 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:42:37.886605    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:43:06.396452 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:06.396413    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"]
Nov 03 16:43:06.396986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:06.396676    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" containerID="cri-o://ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf" gracePeriod=30
Nov 03 16:43:07.889284 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:07.889249    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:43:11.518861 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.517129    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 16:43:11.518861 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.517242    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 16:43:11.687110 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.687085    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:43:11.820037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820007    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-data\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820084    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istiod-ca-cert\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820149    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-podinfo\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820192    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-certs\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820227    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-credential-socket\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820322    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-data" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:11.820444 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820349    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-token\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-token\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820444 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820418    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-envoy\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820546 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820465    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-socket\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820546 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820477    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-certs" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:11.820546 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820521    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-kube-api-access-2wqq5\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-kube-api-access-2wqq5\") pod \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\" (UID: \"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7\") "
Nov 03 16:43:11.820546 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820537    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istiod-ca-cert" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:43:11.820762 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820568    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-credential-socket" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:11.820821 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.820778    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-socket" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:11.821212 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.821175    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.821330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.821243    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.821330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.821260    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.821330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.821275    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.821330 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.821290    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.822483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.822459    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-podinfo" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:43:11.822483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.822471    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-token" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:11.822620 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.822535    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-envoy" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:11.822863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.822848    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-kube-api-access-2wqq5" pod "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" (UID: "83c48fd3-9c73-415d-aec3-a2a9fc42d2a7"). InnerVolumeSpecName "kube-api-access-2wqq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:11.923239 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.923134    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.923239 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.923174    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.923239 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.923187    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:11.923239 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:11.923215    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqq5\" (UniqueName: \"kubernetes.io/projected/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7-kube-api-access-2wqq5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:12.066348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.066313    5112 generic.go:356] "Generic (PLEG): container finished" podID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerID="ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf" exitCode=0
Nov 03 16:43:12.066532 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.066366    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"
Nov 03 16:43:12.066532 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.066390    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" event={"ID":"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7","Type":"ContainerDied","Data":"ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf"}
Nov 03 16:43:12.066532 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.066455    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx" event={"ID":"83c48fd3-9c73-415d-aec3-a2a9fc42d2a7","Type":"ContainerDied","Data":"6a649fb0e9a8661cdbbcc246c3f5a6443f7152aff2cfa62b1e939fc6b096f47b"}
Nov 03 16:43:12.066532 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.066486    5112 scope.go:122] "RemoveContainer" containerID="ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf"
Nov 03 16:43:12.079351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.079294    5112 scope.go:122] "RemoveContainer" containerID="ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf"
Nov 03 16:43:12.079631 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:12.079607    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf\": container with ID starting with ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf not found: ID does not exist" containerID="ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf"
Nov 03 16:43:12.079696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.079642    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf"} err="failed to get container status \"ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf\": rpc error: code = NotFound desc = could not find container \"ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf\": container with ID starting with ba9a0ff511ec2c437536f590e40be499129358da8aed5adc4ecda6d016a3d1cf not found: ID does not exist"
Nov 03 16:43:12.232059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.231991    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"]
Nov 03 16:43:12.239129 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.239106    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-g6xbx"]
Nov 03 16:43:12.551963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:12.551884    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" path="/var/lib/kubelet/pods/83c48fd3-9c73-415d-aec3-a2a9fc42d2a7/volumes"
Nov 03 16:43:20.155287 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:20.155253    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"]
Nov 03 16:43:20.155802 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:20.155534    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy" containerID="cri-o://7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb" gracePeriod=30
Nov 03 16:43:20.464482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:20.464387    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"]
Nov 03 16:43:20.464695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:20.464646    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy" containerID="cri-o://602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922" gracePeriod=30
Nov 03 16:43:20.778020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:20.777941    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:43:20.778020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:20.778001    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:43:21.057893 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:21.057791    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"]
Nov 03 16:43:21.058126 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:21.058092    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" containerID="cri-o://89485c187acd2d3e5c32ea43655c63920846b17502b518e2d9fba3aa269bcca0" gracePeriod=30
Nov 03 16:43:21.890172 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:21.890138    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"]
Nov 03 16:43:21.890646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:21.890426    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy" containerID="cri-o://bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3" gracePeriod=30
Nov 03 16:43:25.119152 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.119118    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:43:25.119631 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.119192    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:43:25.451359 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.451334    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:43:25.526024 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526000    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-credential-socket\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526130 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526048    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-certs\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526130 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526092    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-kube-api-access-hr2xh\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-kube-api-access-hr2xh\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526130 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526122    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-podinfo\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526141    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-token\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-token\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526183    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e248dc-34a8-4b70-9e9b-660296a68e80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e248dc-34a8-4b70-9e9b-660296a68e80-istiod-ca-cert\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526375 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526298    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-envoy\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526375 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526296    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-credential-socket" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.526375 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526351    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-socket\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526390    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-certs" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.526638 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526615    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-data\") pod \"d0e248dc-34a8-4b70-9e9b-660296a68e80\" (UID: \"d0e248dc-34a8-4b70-9e9b-660296a68e80\") "
Nov 03 16:43:25.526734 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526701    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-socket" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.526800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526748    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e248dc-34a8-4b70-9e9b-660296a68e80-istiod-ca-cert" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:43:25.527028 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.526994    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-data" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.527320 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.527300    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e248dc-34a8-4b70-9e9b-660296a68e80-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.527402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.527325    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.527402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.527340    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.527402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.527352    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.527402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.527365    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.528572 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.528551    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-podinfo" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:43:25.528645 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.528609    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-token" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:25.528816 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.528798    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-envoy" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.528859 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.528828    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-kube-api-access-hr2xh" pod "d0e248dc-34a8-4b70-9e9b-660296a68e80" (UID: "d0e248dc-34a8-4b70-9e9b-660296a68e80"). InnerVolumeSpecName "kube-api-access-hr2xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:25.628930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.628898    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr2xh\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-kube-api-access-hr2xh\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.628930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.628925    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.628930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.628935    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.629219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.628959    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e248dc-34a8-4b70-9e9b-660296a68e80-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.756595 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.756569    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:43:25.831431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831403    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-kube-api-access-k97rq\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-kube-api-access-k97rq\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.831605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831448    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-data\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.831605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831523    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-certs\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.831605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831568    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-socket\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.831605 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831604    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/265f1668-3643-4128-b4c9-a1ca831689ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265f1668-3643-4128-b4c9-a1ca831689ed-istio-podinfo\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.831839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831646    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-istio-token\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-istio-token\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.831839 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831821    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265f1668-3643-4128-b4c9-a1ca831689ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265f1668-3643-4128-b4c9-a1ca831689ed-istiod-ca-cert\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.831952 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831876    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-envoy\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.832005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831819    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-data" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.832005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831958    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-credential-socket\") pod \"265f1668-3643-4128-b4c9-a1ca831689ed\" (UID: \"265f1668-3643-4128-b4c9-a1ca831689ed\") "
Nov 03 16:43:25.832134 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831935    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-certs" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.832134 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.831947    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-socket" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.832261 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.832159    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265f1668-3643-4128-b4c9-a1ca831689ed-istiod-ca-cert" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:43:25.832261 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.832212    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-credential-socket" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.832774 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.832748    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265f1668-3643-4128-b4c9-a1ca831689ed-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.832774 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.832772    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.832774 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.832781    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.833050 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.832791    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.833050 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.832802    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.834109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.834082    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-istio-token" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:25.834109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.834085    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-envoy" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:25.834270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.834243    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-kube-api-access-k97rq" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "kube-api-access-k97rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:25.834404 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.834384    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/265f1668-3643-4128-b4c9-a1ca831689ed-istio-podinfo" pod "265f1668-3643-4128-b4c9-a1ca831689ed" (UID: "265f1668-3643-4128-b4c9-a1ca831689ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:43:25.934527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.934489    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k97rq\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-kube-api-access-k97rq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.934527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.934525    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265f1668-3643-4128-b4c9-a1ca831689ed-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.934776 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.934541    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265f1668-3643-4128-b4c9-a1ca831689ed-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:25.934776 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:25.934555    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265f1668-3643-4128-b4c9-a1ca831689ed-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.194960 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.194929    5112 generic.go:356] "Generic (PLEG): container finished" podID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerID="89485c187acd2d3e5c32ea43655c63920846b17502b518e2d9fba3aa269bcca0" exitCode=0
Nov 03 16:43:26.195389 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.195016    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" event={"ID":"cae2b292-a6a2-46e0-bcec-3027cdcd23d4","Type":"ContainerDied","Data":"89485c187acd2d3e5c32ea43655c63920846b17502b518e2d9fba3aa269bcca0"}
Nov 03 16:43:26.196977 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.196942    5112 generic.go:356] "Generic (PLEG): container finished" podID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerID="7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb" exitCode=0
Nov 03 16:43:26.197094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.196995    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"
Nov 03 16:43:26.197094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.197009    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" event={"ID":"d0e248dc-34a8-4b70-9e9b-660296a68e80","Type":"ContainerDied","Data":"7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb"}
Nov 03 16:43:26.197094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.197076    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr" event={"ID":"d0e248dc-34a8-4b70-9e9b-660296a68e80","Type":"ContainerDied","Data":"57db5c65171adabb4359934d6a58f570f6dc7946e9a3cda5d387fae7a255d70e"}
Nov 03 16:43:26.197318 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.197106    5112 scope.go:122] "RemoveContainer" containerID="7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb"
Nov 03 16:43:26.199023 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.199002    5112 generic.go:356] "Generic (PLEG): container finished" podID="265f1668-3643-4128-b4c9-a1ca831689ed" containerID="602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922" exitCode=0
Nov 03 16:43:26.199251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.199192    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" event={"ID":"265f1668-3643-4128-b4c9-a1ca831689ed","Type":"ContainerDied","Data":"602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922"}
Nov 03 16:43:26.199341 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.199281    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w" event={"ID":"265f1668-3643-4128-b4c9-a1ca831689ed","Type":"ContainerDied","Data":"c67229299303cf1d7f3c6264df8aa697da3fb5ccb51e28ef854fba2509863367"}
Nov 03 16:43:26.199405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.199359    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"
Nov 03 16:43:26.211338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.211284    5112 scope.go:122] "RemoveContainer" containerID="7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb"
Nov 03 16:43:26.211589 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:26.211563    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb\": container with ID starting with 7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb not found: ID does not exist" containerID="7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb"
Nov 03 16:43:26.211676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.211596    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb"} err="failed to get container status \"7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb\": rpc error: code = NotFound desc = could not find container \"7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb\": container with ID starting with 7350d37514b389667e9c54e81c40082b76acbcd7f41e75c96e4a74961cc58aeb not found: ID does not exist"
Nov 03 16:43:26.211676 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.211620    5112 scope.go:122] "RemoveContainer" containerID="602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922"
Nov 03 16:43:26.222566 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.222551    5112 scope.go:122] "RemoveContainer" containerID="602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922"
Nov 03 16:43:26.222820 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:26.222801    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922\": container with ID starting with 602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922 not found: ID does not exist" containerID="602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922"
Nov 03 16:43:26.222889 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.222826    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922"} err="failed to get container status \"602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922\": rpc error: code = NotFound desc = could not find container \"602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922\": container with ID starting with 602290c5fd866b960a1af73b945c8449765ab87e023966d0bb5b630aeec4a922 not found: ID does not exist"
Nov 03 16:43:26.362961 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.362932    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"]
Nov 03 16:43:26.372303 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.372278    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:43:26.407619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.407591    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-97q9w"]
Nov 03 16:43:26.441387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441361    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-podinfo\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.441550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441441    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-token\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-token\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.441609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441539    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istiod-ca-cert\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.441688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441659    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-socket\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.441765 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441742    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-kube-api-access-vh2nr\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-kube-api-access-vh2nr\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.441820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441792    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-credential-socket\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.441916 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441901    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-data\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.441974 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441940    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-certs\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.442029 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441975    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-envoy\") pod \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\" (UID: \"cae2b292-a6a2-46e0-bcec-3027cdcd23d4\") "
Nov 03 16:43:26.442821 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441973    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-socket" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:26.442945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.441982    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istiod-ca-cert" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:43:26.442945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.442192    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-data" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:26.442945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.442410    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-credential-socket" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:26.442945 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.442587    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-certs" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:26.443880 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.443854    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-podinfo" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:43:26.444169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.444140    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-token" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:26.444423 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.444396    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-kube-api-access-vh2nr" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "kube-api-access-vh2nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:26.444776 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.444755    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-envoy" pod "cae2b292-a6a2-46e0-bcec-3027cdcd23d4" (UID: "cae2b292-a6a2-46e0-bcec-3027cdcd23d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:26.472567 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.472545    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"]
Nov 03 16:43:26.479443 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.479419    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-dkxsr"]
Nov 03 16:43:26.543517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543493    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543516    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543527    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543536    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543546    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2nr\" (UniqueName: \"kubernetes.io/projected/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-kube-api-access-vh2nr\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543555    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543566    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543575    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.543683 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.543582    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae2b292-a6a2-46e0-bcec-3027cdcd23d4-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:26.551508 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.551474    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" path="/var/lib/kubelet/pods/265f1668-3643-4128-b4c9-a1ca831689ed/volumes"
Nov 03 16:43:26.551992 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:26.551978    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" path="/var/lib/kubelet/pods/d0e248dc-34a8-4b70-9e9b-660296a68e80/volumes"
Nov 03 16:43:27.163503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.163428    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:43:27.209239 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.209187    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"
Nov 03 16:43:27.209239 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.209225    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm" event={"ID":"cae2b292-a6a2-46e0-bcec-3027cdcd23d4","Type":"ContainerDied","Data":"ebbe6d1e30daca56a71fbcaa8d7bbebbbccd35b39b7bf155ff068ea81a86d865"}
Nov 03 16:43:27.209659 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.209291    5112 scope.go:122] "RemoveContainer" containerID="89485c187acd2d3e5c32ea43655c63920846b17502b518e2d9fba3aa269bcca0"
Nov 03 16:43:27.210801 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.210768    5112 generic.go:356] "Generic (PLEG): container finished" podID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerID="bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3" exitCode=0
Nov 03 16:43:27.210922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.210820    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"
Nov 03 16:43:27.210922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.210849    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" event={"ID":"6f59c3b3-743c-4afe-8efe-dc5dc906a495","Type":"ContainerDied","Data":"bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3"}
Nov 03 16:43:27.210922 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.210901    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr" event={"ID":"6f59c3b3-743c-4afe-8efe-dc5dc906a495","Type":"ContainerDied","Data":"e71be1d3eea61c71e4a48b53c5aca6f4c759bb3d834304a41915069d9ebc6ce3"}
Nov 03 16:43:27.220245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.220230    5112 scope.go:122] "RemoveContainer" containerID="bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3"
Nov 03 16:43:27.230624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.230599    5112 scope.go:122] "RemoveContainer" containerID="bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3"
Nov 03 16:43:27.230918 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:27.230900    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3\": container with ID starting with bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3 not found: ID does not exist" containerID="bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3"
Nov 03 16:43:27.231025 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.230929    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3"} err="failed to get container status \"bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3\": rpc error: code = NotFound desc = could not find container \"bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3\": container with ID starting with bdb0a3674c3985d42564bd5d3c253e461f2b146917cddd5cdf1c7a05683a3ef3 not found: ID does not exist"
Nov 03 16:43:27.254327 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254296    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-credential-socket\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254357    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-socket\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254410    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istiod-ca-cert\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254633 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254499    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-podinfo\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254633 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254544    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-kube-api-access-brrl4\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-kube-api-access-brrl4\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254743 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254630    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-credential-socket" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:27.254743 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254708    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-data\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254844 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254755    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-certs\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254844 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254827    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-envoy\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.254949 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.254860    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istiod-ca-cert" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:43:27.255038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.255018    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-token\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-token\") pod \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\" (UID: \"6f59c3b3-743c-4afe-8efe-dc5dc906a495\") "
Nov 03 16:43:27.255431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.255104    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-data" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:27.255743 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.255700    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.255743 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.255723    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.255743 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.255738    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.256005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.255982    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-certs" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:27.256244 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.256188    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-socket" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:27.258792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.258739    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-kube-api-access-brrl4" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "kube-api-access-brrl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:27.258900 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.258811    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-envoy" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:43:27.259276 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.259253    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-token" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:27.259276 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.259259    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-podinfo" pod "6f59c3b3-743c-4afe-8efe-dc5dc906a495" (UID: "6f59c3b3-743c-4afe-8efe-dc5dc906a495"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:43:27.358730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.358690    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.358730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.358730    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.358935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.358747    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.358935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.358762    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brrl4\" (UniqueName: \"kubernetes.io/projected/6f59c3b3-743c-4afe-8efe-dc5dc906a495-kube-api-access-brrl4\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.358935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.358781    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.358935 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.358797    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f59c3b3-743c-4afe-8efe-dc5dc906a495-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:27.400356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.400328    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"]
Nov 03 16:43:27.404150 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.404124    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-78wfm"]
Nov 03 16:43:27.636837 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.636801    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"]
Nov 03 16:43:27.642057 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:27.642028    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-zqkxr"]
Nov 03 16:43:28.552040 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:28.552008    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" path="/var/lib/kubelet/pods/6f59c3b3-743c-4afe-8efe-dc5dc906a495/volumes"
Nov 03 16:43:28.552570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:28.552553    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" path="/var/lib/kubelet/pods/cae2b292-a6a2-46e0-bcec-3027cdcd23d4/volumes"
Nov 03 16:43:37.884517 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:37.884481    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:43:45.225774 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.225740    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7133-d3dcb2"]
Nov 03 16:43:45.226474 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226459    5112 cpu_manager.go:414] "containerMap: removing container" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy"
Nov 03 16:43:45.226527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226476    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy"
Nov 03 16:43:45.226527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226488    5112 cpu_manager.go:414] "containerMap: removing container" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy"
Nov 03 16:43:45.226527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226493    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy"
Nov 03 16:43:45.226527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226508    5112 cpu_manager.go:414] "containerMap: removing container" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy"
Nov 03 16:43:45.226527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226513    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy"
Nov 03 16:43:45.226527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226519    5112 cpu_manager.go:414] "containerMap: removing container" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy"
Nov 03 16:43:45.226527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226523    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226539    5112 cpu_manager.go:414] "containerMap: removing container" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226545    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226556    5112 cpu_manager.go:414] "containerMap: removing container" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226561    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226569    5112 cpu_manager.go:414] "containerMap: removing container" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226573    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226580    5112 cpu_manager.go:414] "containerMap: removing container" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226584    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226712    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e248dc-34a8-4b70-9e9b-660296a68e80" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226727    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae2b292-a6a2-46e0-bcec-3027cdcd23d4" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226739    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="40eb21d3-2ad2-4c2c-a22c-7867a2352537" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226750    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="487f67a9-7a18-4994-800c-66c7690aff43" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226765    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f59c3b3-743c-4afe-8efe-dc5dc906a495" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226772    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="441034a0-b050-4421-8294-e6401e9049c4" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226778    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="265f1668-3643-4128-b4c9-a1ca831689ed" containerName="istio-proxy"
Nov 03 16:43:45.226784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.226784    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c48fd3-9c73-415d-aec3-a2a9fc42d2a7" containerName="istio-proxy"
Nov 03 16:43:45.231540 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.231518    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:45.282118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.282082    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-d3dcb2"]
Nov 03 16:43:45.324720 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.324693    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8nh5\""
Nov 03 16:43:45.369819 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.369792    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq4pr\" (UniqueName: \"kubernetes.io/projected/88c28f33-828e-4b02-88fb-d57847496715-kube-api-access-pq4pr\") pod \"test-curl-7133-d3dcb2\" (UID: \"88c28f33-828e-4b02-88fb-d57847496715\") " pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:45.471395 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.471368    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq4pr\" (UniqueName: \"kubernetes.io/projected/88c28f33-828e-4b02-88fb-d57847496715-kube-api-access-pq4pr\") pod \"test-curl-7133-d3dcb2\" (UID: \"88c28f33-828e-4b02-88fb-d57847496715\") " pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:45.527409 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.527347    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq4pr\" (UniqueName: \"kubernetes.io/projected/88c28f33-828e-4b02-88fb-d57847496715-kube-api-access-pq4pr\") pod \"test-curl-7133-d3dcb2\" (UID: \"88c28f33-828e-4b02-88fb-d57847496715\") " pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:45.544346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.544323    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:45.931603 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.931570    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-d3dcb2"]
Nov 03 16:43:45.932031 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:43:45.932005    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c28f33_828e_4b02_88fb_d57847496715.slice/crio-0c7b52970b46ee7f500b7e2b63ab6edb1c4c66d6a9b541f865a82b5fd2545568 WatchSource:0}: Error finding container 0c7b52970b46ee7f500b7e2b63ab6edb1c4c66d6a9b541f865a82b5fd2545568: Status 404 returned error can't find the container with id 0c7b52970b46ee7f500b7e2b63ab6edb1c4c66d6a9b541f865a82b5fd2545568
Nov 03 16:43:45.933784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:45.933767    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:43:46.374270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:46.374165    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-d3dcb2" event={"ID":"88c28f33-828e-4b02-88fb-d57847496715","Type":"ContainerStarted","Data":"0c7b52970b46ee7f500b7e2b63ab6edb1c4c66d6a9b541f865a82b5fd2545568"}
Nov 03 16:43:48.402867 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:48.402824    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-d3dcb2" event={"ID":"88c28f33-828e-4b02-88fb-d57847496715","Type":"ContainerStarted","Data":"2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5"}
Nov 03 16:43:48.512742 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:48.512678    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7133-d3dcb2" podStartSLOduration=1.308597533 podStartE2EDuration="3.512659885s" totalImagesPullingTime="2.204062352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:48.511877675 +0000 UTC m=+1990.091181553" watchObservedRunningTime="2026-11-03 16:43:48.512659885 +0000 UTC m=+1990.091963741"
Nov 03 16:43:49.966817 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:49.966781    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7133-d3dcb2"]
Nov 03 16:43:49.979359 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:49.979331    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7133-d3dcb2"]
Nov 03 16:43:50.135786 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.135744    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7133-5d5765"]
Nov 03 16:43:50.136518 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.136502    5112 cpu_manager.go:414] "containerMap: removing container" podUID="88c28f33-828e-4b02-88fb-d57847496715" containerName="test-curl-7133-d3dcb2"
Nov 03 16:43:50.136574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.136519    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c28f33-828e-4b02-88fb-d57847496715" containerName="test-curl-7133-d3dcb2"
Nov 03 16:43:50.136650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.136639    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c28f33-828e-4b02-88fb-d57847496715" containerName="test-curl-7133-d3dcb2"
Nov 03 16:43:50.140465 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.140449    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:50.193320 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.193288    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-5d5765"]
Nov 03 16:43:50.237655 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.237582    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbl59\" (UniqueName: \"kubernetes.io/projected/fff3293e-39c5-4086-bddb-e87f79a8a891-kube-api-access-mbl59\") pod \"test-curl-7133-5d5765\" (UID: \"fff3293e-39c5-4086-bddb-e87f79a8a891\") " pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:50.339354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.339327    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbl59\" (UniqueName: \"kubernetes.io/projected/fff3293e-39c5-4086-bddb-e87f79a8a891-kube-api-access-mbl59\") pod \"test-curl-7133-5d5765\" (UID: \"fff3293e-39c5-4086-bddb-e87f79a8a891\") " pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:50.395953 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.395918    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbl59\" (UniqueName: \"kubernetes.io/projected/fff3293e-39c5-4086-bddb-e87f79a8a891-kube-api-access-mbl59\") pod \"test-curl-7133-5d5765\" (UID: \"fff3293e-39c5-4086-bddb-e87f79a8a891\") " pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:50.418185 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.418152    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7133-d3dcb2" podUID="88c28f33-828e-4b02-88fb-d57847496715" containerName="test-curl-7133-d3dcb2" containerID="cri-o://2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5" gracePeriod=2
Nov 03 16:43:50.452973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.452950    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:50.467257 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:50.467022    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-d3dcb2\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="88c28f33-828e-4b02-88fb-d57847496715" pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:50.841761 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:43:50.841712    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfff3293e_39c5_4086_bddb_e87f79a8a891.slice/crio-02f8a8928902bc82f1c75fb7d35fa07664498377f4a6eb931ba0951f663c0dae WatchSource:0}: Error finding container 02f8a8928902bc82f1c75fb7d35fa07664498377f4a6eb931ba0951f663c0dae: Status 404 returned error can't find the container with id 02f8a8928902bc82f1c75fb7d35fa07664498377f4a6eb931ba0951f663c0dae
Nov 03 16:43:50.841899 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:50.841823    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-5d5765"]
Nov 03 16:43:51.426579 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:51.426543    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-5d5765" event={"ID":"fff3293e-39c5-4086-bddb-e87f79a8a891","Type":"ContainerStarted","Data":"a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71"}
Nov 03 16:43:51.426579 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:51.426590    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-5d5765" event={"ID":"fff3293e-39c5-4086-bddb-e87f79a8a891","Type":"ContainerStarted","Data":"02f8a8928902bc82f1c75fb7d35fa07664498377f4a6eb931ba0951f663c0dae"}
Nov 03 16:43:51.537534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:51.537481    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7133-5d5765" podStartSLOduration=1.5374657 podStartE2EDuration="1.5374657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:51.536564694 +0000 UTC m=+1993.115868550" watchObservedRunningTime="2026-11-03 16:43:51.5374657 +0000 UTC m=+1993.116769555"
Nov 03 16:43:52.496957 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:52.496859    5112 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c28f33_828e_4b02_88fb_d57847496715.slice/crio-conmon-2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c28f33_828e_4b02_88fb_d57847496715.slice/crio-2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:43:52.734269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:52.734245    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:52.773123 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:52.773093    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c28f33-828e-4b02-88fb-d57847496715-kube-api-access-pq4pr\" (UniqueName: \"kubernetes.io/projected/88c28f33-828e-4b02-88fb-d57847496715-kube-api-access-pq4pr\") pod \"88c28f33-828e-4b02-88fb-d57847496715\" (UID: \"88c28f33-828e-4b02-88fb-d57847496715\") "
Nov 03 16:43:52.775168 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:52.775141    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c28f33-828e-4b02-88fb-d57847496715-kube-api-access-pq4pr" pod "88c28f33-828e-4b02-88fb-d57847496715" (UID: "88c28f33-828e-4b02-88fb-d57847496715"). InnerVolumeSpecName "kube-api-access-pq4pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:52.874968 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:52.874934    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq4pr\" (UniqueName: \"kubernetes.io/projected/88c28f33-828e-4b02-88fb-d57847496715-kube-api-access-pq4pr\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:53.112513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.112420    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7133-5d5765"]
Nov 03 16:43:53.117907 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.117879    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7133-5d5765"]
Nov 03 16:43:53.276868 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.276833    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7133-371e73"]
Nov 03 16:43:53.277634 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.277615    5112 cpu_manager.go:414] "containerMap: removing container" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" containerName="test-curl-7133-5d5765"
Nov 03 16:43:53.277634 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.277634    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" containerName="test-curl-7133-5d5765"
Nov 03 16:43:53.277772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.277752    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" containerName="test-curl-7133-5d5765"
Nov 03 16:43:53.281733 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.281715    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:53.334756 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.334731    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-371e73"]
Nov 03 16:43:53.381257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.381168    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sph4\" (UniqueName: \"kubernetes.io/projected/83daca46-2bbf-476c-87c9-fa8ac4403858-kube-api-access-2sph4\") pod \"test-curl-7133-371e73\" (UID: \"83daca46-2bbf-476c-87c9-fa8ac4403858\") " pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:53.482886 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.482850    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sph4\" (UniqueName: \"kubernetes.io/projected/83daca46-2bbf-476c-87c9-fa8ac4403858-kube-api-access-2sph4\") pod \"test-curl-7133-371e73\" (UID: \"83daca46-2bbf-476c-87c9-fa8ac4403858\") " pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:53.522525 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.522492    5112 generic.go:356] "Generic (PLEG): container finished" podID="88c28f33-828e-4b02-88fb-d57847496715" containerID="2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5" exitCode=137
Nov 03 16:43:53.522881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.522533    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:53.522881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.522586    5112 scope.go:122] "RemoveContainer" containerID="2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5"
Nov 03 16:43:53.522881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.522826    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7133-5d5765" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" containerName="test-curl-7133-5d5765" containerID="cri-o://a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71" gracePeriod=2
Nov 03 16:43:53.535704 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.535683    5112 scope.go:122] "RemoveContainer" containerID="2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5"
Nov 03 16:43:53.535977 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:53.535959    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5\": container with ID starting with 2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5 not found: ID does not exist" containerID="2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5"
Nov 03 16:43:53.536031 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.535986    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5"} err="failed to get container status \"2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5\": rpc error: code = NotFound desc = could not find container \"2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5\": container with ID starting with 2524799c48400c386d12c7133f8b3a099a76eb0a5acf6341dde479e61ac758f5 not found: ID does not exist"
Nov 03 16:43:53.538929 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.538889    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sph4\" (UniqueName: \"kubernetes.io/projected/83daca46-2bbf-476c-87c9-fa8ac4403858-kube-api-access-2sph4\") pod \"test-curl-7133-371e73\" (UID: \"83daca46-2bbf-476c-87c9-fa8ac4403858\") " pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:53.571535 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:53.571499    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-5d5765\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:53.597543 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.597521    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:53.620046 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:53.620015    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-5d5765\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:53.681110 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:53.668918    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-d3dcb2\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="88c28f33-828e-4b02-88fb-d57847496715" pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:53.980984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:53.980950    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-371e73"]
Nov 03 16:43:53.981252 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:43:53.981068    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83daca46_2bbf_476c_87c9_fa8ac4403858.slice/crio-cbb9ecc4da6edeae7b764b32da288fbbfc46fc5cc6faedd4be817692d73a2fce WatchSource:0}: Error finding container cbb9ecc4da6edeae7b764b32da288fbbfc46fc5cc6faedd4be817692d73a2fce: Status 404 returned error can't find the container with id cbb9ecc4da6edeae7b764b32da288fbbfc46fc5cc6faedd4be817692d73a2fce
Nov 03 16:43:54.532152 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:54.532102    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-371e73" event={"ID":"83daca46-2bbf-476c-87c9-fa8ac4403858","Type":"ContainerStarted","Data":"73314fc67308388c53a90a1c7f783850e831f9e0b4eaa4b67535661edf78d1f6"}
Nov 03 16:43:54.532566 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:54.532170    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-371e73" event={"ID":"83daca46-2bbf-476c-87c9-fa8ac4403858","Type":"ContainerStarted","Data":"cbb9ecc4da6edeae7b764b32da288fbbfc46fc5cc6faedd4be817692d73a2fce"}
Nov 03 16:43:54.551189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:54.551163    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c28f33-828e-4b02-88fb-d57847496715" path="/var/lib/kubelet/pods/88c28f33-828e-4b02-88fb-d57847496715/volumes"
Nov 03 16:43:54.644514 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:54.644454    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7133-371e73" podStartSLOduration=1.644437065 podStartE2EDuration="1.644437065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:54.641147546 +0000 UTC m=+1996.220451419" watchObservedRunningTime="2026-11-03 16:43:54.644437065 +0000 UTC m=+1996.223740932"
Nov 03 16:43:54.689369 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:54.689318    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-5d5765\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:54.737556 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:54.737522    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-d3dcb2\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="88c28f33-828e-4b02-88fb-d57847496715" pod="opendatahub/test-curl-7133-d3dcb2"
Nov 03 16:43:55.764109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:55.764078    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:55.843082 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:55.843013    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff3293e-39c5-4086-bddb-e87f79a8a891-kube-api-access-mbl59\" (UniqueName: \"kubernetes.io/projected/fff3293e-39c5-4086-bddb-e87f79a8a891-kube-api-access-mbl59\") pod \"fff3293e-39c5-4086-bddb-e87f79a8a891\" (UID: \"fff3293e-39c5-4086-bddb-e87f79a8a891\") "
Nov 03 16:43:55.845189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:55.845164    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff3293e-39c5-4086-bddb-e87f79a8a891-kube-api-access-mbl59" pod "fff3293e-39c5-4086-bddb-e87f79a8a891" (UID: "fff3293e-39c5-4086-bddb-e87f79a8a891"). InnerVolumeSpecName "kube-api-access-mbl59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:55.945075 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:55.945046    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbl59\" (UniqueName: \"kubernetes.io/projected/fff3293e-39c5-4086-bddb-e87f79a8a891-kube-api-access-mbl59\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:56.193490 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.193461    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7133-371e73"]
Nov 03 16:43:56.203521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.203488    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7133-371e73"]
Nov 03 16:43:56.347792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.347758    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7133-2e4b7c"]
Nov 03 16:43:56.348901 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.348876    5112 cpu_manager.go:414] "containerMap: removing container" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" containerName="test-curl-7133-371e73"
Nov 03 16:43:56.348999 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.348906    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" containerName="test-curl-7133-371e73"
Nov 03 16:43:56.349118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.349104    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" containerName="test-curl-7133-371e73"
Nov 03 16:43:56.353125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.353106    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:43:56.405033 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.405006    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-2e4b7c"]
Nov 03 16:43:56.453094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.453022    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnpj\" (UniqueName: \"kubernetes.io/projected/8af60878-299e-4809-af9f-34187302f829-kube-api-access-nvnpj\") pod \"test-curl-7133-2e4b7c\" (UID: \"8af60878-299e-4809-af9f-34187302f829\") " pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:43:56.560348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.556080    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnpj\" (UniqueName: \"kubernetes.io/projected/8af60878-299e-4809-af9f-34187302f829-kube-api-access-nvnpj\") pod \"test-curl-7133-2e4b7c\" (UID: \"8af60878-299e-4809-af9f-34187302f829\") " pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:43:56.630325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.630293    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnpj\" (UniqueName: \"kubernetes.io/projected/8af60878-299e-4809-af9f-34187302f829-kube-api-access-nvnpj\") pod \"test-curl-7133-2e4b7c\" (UID: \"8af60878-299e-4809-af9f-34187302f829\") " pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:43:56.636717 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.636665    5112 generic.go:356] "Generic (PLEG): container finished" podID="fff3293e-39c5-4086-bddb-e87f79a8a891" containerID="a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71" exitCode=137
Nov 03 16:43:56.636970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.636944    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-5d5765"
Nov 03 16:43:56.637057 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.637033    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7133-371e73" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" containerName="test-curl-7133-371e73" containerID="cri-o://73314fc67308388c53a90a1c7f783850e831f9e0b4eaa4b67535661edf78d1f6" gracePeriod=2
Nov 03 16:43:56.642178 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.642147    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff3293e-39c5-4086-bddb-e87f79a8a891" path="/var/lib/kubelet/pods/fff3293e-39c5-4086-bddb-e87f79a8a891/volumes"
Nov 03 16:43:56.642684 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.642667    5112 scope.go:122] "RemoveContainer" containerID="a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71"
Nov 03 16:43:56.659607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.659585    5112 scope.go:122] "RemoveContainer" containerID="a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71"
Nov 03 16:43:56.660176 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:56.660144    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71\": container with ID starting with a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71 not found: ID does not exist" containerID="a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71"
Nov 03 16:43:56.660363 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.660184    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71"} err="failed to get container status \"a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71\": rpc error: code = NotFound desc = could not find container \"a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71\": container with ID starting with a1f7c345eea0b3e0587ed8d8eb62f80b2c040e64e9fd2a3667040312bc653e71 not found: ID does not exist"
Nov 03 16:43:56.668272 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.668250    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:43:56.750837 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:56.750798    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-371e73\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:56.853943 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.853567    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"]
Nov 03 16:43:56.863433 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.863390    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:56.914240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.913653    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"]
Nov 03 16:43:56.916770 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.916720    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0ea8a5\""
Nov 03 16:43:56.968051 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.968021    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fpt\" (UniqueName: \"kubernetes.io/projected/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-kube-api-access-c6fpt\") pod \"payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:56.968292 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:56.968241    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:57.074374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.070147    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fpt\" (UniqueName: \"kubernetes.io/projected/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-kube-api-access-c6fpt\") pod \"payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:57.074374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.070380    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:57.074374 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.071554    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:57.199236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.198972    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fpt\" (UniqueName: \"kubernetes.io/projected/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-kube-api-access-c6fpt\") pod \"payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:57.277704 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.277674    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7133-2e4b7c"]
Nov 03 16:43:57.487399 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.487368    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:57.581347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.580804    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"]
Nov 03 16:43:57.588706 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.587810    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.640040 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.640003    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0fda73-serving-cert\""
Nov 03 16:43:57.680179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.680146    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"]
Nov 03 16:43:57.684827 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.684797    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-2e4b7c" event={"ID":"8af60878-299e-4809-af9f-34187302f829","Type":"ContainerStarted","Data":"89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04"}
Nov 03 16:43:57.685007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.684988    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7133-2e4b7c" event={"ID":"8af60878-299e-4809-af9f-34187302f829","Type":"ContainerStarted","Data":"f9b9049f2b1cb272db62d66d1874ad045692207d09c8772cca2a89203b21198b"}
Nov 03 16:43:57.685623 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.685336    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8677da00-1571-4dbf-ad73-6432017e538e-openshift-service-ca\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.685623 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.685409    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-metrics-certs\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.685810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.685625    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvh2\" (UniqueName: \"kubernetes.io/projected/8677da00-1571-4dbf-ad73-6432017e538e-kube-api-access-sqvh2\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.685810 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.685704    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.786941 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.786904    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvh2\" (UniqueName: \"kubernetes.io/projected/8677da00-1571-4dbf-ad73-6432017e538e-kube-api-access-sqvh2\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.787128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.786972    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.787128 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:57.787068    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-serving-cert: secret "maas-api-e2e-api-b-0fda73-serving-cert" not found
Nov 03 16:43:57.787128 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.787094    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8677da00-1571-4dbf-ad73-6432017e538e-openshift-service-ca\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.787331 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:57.787137    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls podName:8677da00-1571-4dbf-ad73-6432017e538e nodeName:}" failed. No retries permitted until 2026-11-03 16:43:58.287114656 +0000 UTC m=+1999.866418506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls") pod "maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk" (UID: "8677da00-1571-4dbf-ad73-6432017e538e") : secret "maas-api-e2e-api-b-0fda73-serving-cert" not found
Nov 03 16:43:57.787432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.787409    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-metrics-certs\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.787910 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.787882    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8677da00-1571-4dbf-ad73-6432017e538e-openshift-service-ca\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.790001 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.789977    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-metrics-certs\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.836289 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.835131    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7133-2e4b7c" podStartSLOduration=1.83510997 podStartE2EDuration="1.83510997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:57.816747582 +0000 UTC m=+1999.396051460" watchObservedRunningTime="2026-11-03 16:43:57.83510997 +0000 UTC m=+1999.414413825"
Nov 03 16:43:57.869551 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.868285    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvh2\" (UniqueName: \"kubernetes.io/projected/8677da00-1571-4dbf-ad73-6432017e538e-kube-api-access-sqvh2\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:57.879694 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:57.879665    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"]
Nov 03 16:43:57.882332 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:43:57.882302    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37bc61b_61dc_4b3b_94a3_bbc54babfca8.slice/crio-7ff587123462b0fca1b16685cd79a419e2dacbd4b3b6a68fa1b6ab8336532957 WatchSource:0}: Error finding container 7ff587123462b0fca1b16685cd79a419e2dacbd4b3b6a68fa1b6ab8336532957: Status 404 returned error can't find the container with id 7ff587123462b0fca1b16685cd79a419e2dacbd4b3b6a68fa1b6ab8336532957
Nov 03 16:43:58.300214 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.300170    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:58.302864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.302834    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls\") pod \"maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:58.517225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.516873    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:43:58.697672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.697491    5112 generic.go:356] "Generic (PLEG): container finished" podID="83daca46-2bbf-476c-87c9-fa8ac4403858" containerID="73314fc67308388c53a90a1c7f783850e831f9e0b4eaa4b67535661edf78d1f6" exitCode=137
Nov 03 16:43:58.701027 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.700998    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4" event={"ID":"d37bc61b-61dc-4b3b-94a3-bbc54babfca8","Type":"ContainerStarted","Data":"d651623c9092b006694f46a3a4c92d7c651992ea51969e777ba8cfbfbe9363f6"}
Nov 03 16:43:58.701393 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.701375    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:43:58.701559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.701539    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4" event={"ID":"d37bc61b-61dc-4b3b-94a3-bbc54babfca8","Type":"ContainerStarted","Data":"7ff587123462b0fca1b16685cd79a419e2dacbd4b3b6a68fa1b6ab8336532957"}
Nov 03 16:43:58.829230 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.826990    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4" podStartSLOduration=2.826971366 podStartE2EDuration="2.826971366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:58.824460179 +0000 UTC m=+2000.403764035" watchObservedRunningTime="2026-11-03 16:43:58.826971366 +0000 UTC m=+2000.406275222"
Nov 03 16:43:58.905664 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.905638    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:58.958837 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:58.958421    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-371e73\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:58.985925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:58.984673    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"]
Nov 03 16:43:59.014486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.014457    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83daca46-2bbf-476c-87c9-fa8ac4403858-kube-api-access-2sph4\" (UniqueName: \"kubernetes.io/projected/83daca46-2bbf-476c-87c9-fa8ac4403858-kube-api-access-2sph4\") pod \"83daca46-2bbf-476c-87c9-fa8ac4403858\" (UID: \"83daca46-2bbf-476c-87c9-fa8ac4403858\") "
Nov 03 16:43:59.016646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.016617    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83daca46-2bbf-476c-87c9-fa8ac4403858-kube-api-access-2sph4" pod "83daca46-2bbf-476c-87c9-fa8ac4403858" (UID: "83daca46-2bbf-476c-87c9-fa8ac4403858"). InnerVolumeSpecName "kube-api-access-2sph4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:43:59.117029 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.116965    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sph4\" (UniqueName: \"kubernetes.io/projected/83daca46-2bbf-476c-87c9-fa8ac4403858-kube-api-access-2sph4\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:43:59.332485 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.332453    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7133-2e4b7c"]
Nov 03 16:43:59.341433 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.341403    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7133-2e4b7c"]
Nov 03 16:43:59.713828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.713803    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:59.713992 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.713855    5112 scope.go:122] "RemoveContainer" containerID="73314fc67308388c53a90a1c7f783850e831f9e0b4eaa4b67535661edf78d1f6"
Nov 03 16:43:59.715561 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.715530    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk" event={"ID":"8677da00-1571-4dbf-ad73-6432017e538e","Type":"ContainerStarted","Data":"55752e861a464c7b43fb3f4f416094d76e69685ae050702b00048041c30bfa66"}
Nov 03 16:43:59.715699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:43:59.715594    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7133-2e4b7c" podUID="8af60878-299e-4809-af9f-34187302f829" containerName="test-curl-7133-2e4b7c" containerID="cri-o://89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04" gracePeriod=2
Nov 03 16:43:59.763639 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:59.763599    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-371e73\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:59.813920 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:59.813882    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-371e73\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" pod="opendatahub/test-curl-7133-371e73"
Nov 03 16:43:59.867224 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:43:59.866555    5112 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7133-2e4b7c\" is forbidden: User \"system:node:ip-10-0-137-122.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-122.us-west-2.compute.internal' and this object" podUID="8af60878-299e-4809-af9f-34187302f829" pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:44:00.575223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:00.574738    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83daca46-2bbf-476c-87c9-fa8ac4403858" path="/var/lib/kubelet/pods/83daca46-2bbf-476c-87c9-fa8ac4403858/volumes"
Nov 03 16:44:00.753189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:00.744294    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk" event={"ID":"8677da00-1571-4dbf-ad73-6432017e538e","Type":"ContainerStarted","Data":"6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef"}
Nov 03 16:44:00.753189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:00.745438    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:44:00.913398 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:00.910107    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk" podStartSLOduration=2.849799168 podStartE2EDuration="3.910084339s" totalImagesPullingTime="1.060285171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:00.882569311 +0000 UTC m=+2002.461873166" watchObservedRunningTime="2026-11-03 16:44:00.910084339 +0000 UTC m=+2002.489388195"
Nov 03 16:44:01.790804 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:44:01.790778    5112 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8af60878_299e_4809_af9f_34187302f829.slice/crio-89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8af60878_299e_4809_af9f_34187302f829.slice/crio-conmon-89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 16:44:02.197268 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.196954    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:44:02.258969 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.258935    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af60878-299e-4809-af9f-34187302f829-kube-api-access-nvnpj\" (UniqueName: \"kubernetes.io/projected/8af60878-299e-4809-af9f-34187302f829-kube-api-access-nvnpj\") pod \"8af60878-299e-4809-af9f-34187302f829\" (UID: \"8af60878-299e-4809-af9f-34187302f829\") "
Nov 03 16:44:02.261173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.261146    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af60878-299e-4809-af9f-34187302f829-kube-api-access-nvnpj" pod "8af60878-299e-4809-af9f-34187302f829" (UID: "8af60878-299e-4809-af9f-34187302f829"). InnerVolumeSpecName "kube-api-access-nvnpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:44:02.370254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.362470    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvnpj\" (UniqueName: \"kubernetes.io/projected/8af60878-299e-4809-af9f-34187302f829-kube-api-access-nvnpj\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:02.551649 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.551613    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af60878-299e-4809-af9f-34187302f829" path="/var/lib/kubelet/pods/8af60878-299e-4809-af9f-34187302f829/volumes"
Nov 03 16:44:02.763053 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.763009    5112 generic.go:356] "Generic (PLEG): container finished" podID="8af60878-299e-4809-af9f-34187302f829" containerID="89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04" exitCode=137
Nov 03 16:44:02.763260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.763072    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7133-2e4b7c"
Nov 03 16:44:02.763260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.763116    5112 scope.go:122] "RemoveContainer" containerID="89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04"
Nov 03 16:44:02.774865 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.774844    5112 scope.go:122] "RemoveContainer" containerID="89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04"
Nov 03 16:44:02.775158 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:44:02.775133    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04\": container with ID starting with 89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04 not found: ID does not exist" containerID="89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04"
Nov 03 16:44:02.775257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:02.775167    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04"} err="failed to get container status \"89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04\": rpc error: code = NotFound desc = could not find container \"89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04\": container with ID starting with 89131f7022aa38bb2e97ab0238cd3f9da378954ccb81523c37f7ff80727d6c04 not found: ID does not exist"
Nov 03 16:44:07.302347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.292188    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"]
Nov 03 16:44:07.302866 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.302428    5112 cpu_manager.go:414] "containerMap: removing container" podUID="8af60878-299e-4809-af9f-34187302f829" containerName="test-curl-7133-2e4b7c"
Nov 03 16:44:07.302866 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.302451    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af60878-299e-4809-af9f-34187302f829" containerName="test-curl-7133-2e4b7c"
Nov 03 16:44:07.302866 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.302635    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af60878-299e-4809-af9f-34187302f829" containerName="test-curl-7133-2e4b7c"
Nov 03 16:44:07.313471 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.310438    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:07.346692 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.346455    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"]
Nov 03 16:44:07.388266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.388181    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6822cb\""
Nov 03 16:44:07.433639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.433599    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6822cb-dockercfg-8l9xh\""
Nov 03 16:44:07.546825 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.546763    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d3d850-986d-4dbf-9854-a8c647328cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:07.547266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.547240    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhncv\" (UniqueName: \"kubernetes.io/projected/c3d3d850-986d-4dbf-9854-a8c647328cb8-kube-api-access-zhncv\") pod \"payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:07.649482 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.649439    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d3d850-986d-4dbf-9854-a8c647328cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:07.649687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.649587    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhncv\" (UniqueName: \"kubernetes.io/projected/c3d3d850-986d-4dbf-9854-a8c647328cb8-kube-api-access-zhncv\") pod \"payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:07.651135 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.651082    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d3d850-986d-4dbf-9854-a8c647328cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:07.708550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.708517    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhncv\" (UniqueName: \"kubernetes.io/projected/c3d3d850-986d-4dbf-9854-a8c647328cb8-kube-api-access-zhncv\") pod \"payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:07.769281 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.769251    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:44:07.884259 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:44:07.884228    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:44:07.932353 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:07.932270    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:08.319360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:08.319319    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"]
Nov 03 16:44:08.320345 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:44:08.320278    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d3d850_986d_4dbf_9854_a8c647328cb8.slice/crio-252785832b923a33251e545be91ee52cbefc615ec9efe21388d29964cbba4fef WatchSource:0}: Error finding container 252785832b923a33251e545be91ee52cbefc615ec9efe21388d29964cbba4fef: Status 404 returned error can't find the container with id 252785832b923a33251e545be91ee52cbefc615ec9efe21388d29964cbba4fef
Nov 03 16:44:08.854775 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:08.854734    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq" event={"ID":"c3d3d850-986d-4dbf-9854-a8c647328cb8","Type":"ContainerStarted","Data":"9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4"}
Nov 03 16:44:08.854982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:08.854803    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq" event={"ID":"c3d3d850-986d-4dbf-9854-a8c647328cb8","Type":"ContainerStarted","Data":"252785832b923a33251e545be91ee52cbefc615ec9efe21388d29964cbba4fef"}
Nov 03 16:44:08.854982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:08.854833    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:08.973512 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:08.973459    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq" podStartSLOduration=1.9734450319999999 podStartE2EDuration="1.973445032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:08.97093951 +0000 UTC m=+2010.550243388" watchObservedRunningTime="2026-11-03 16:44:08.973445032 +0000 UTC m=+2010.552748900"
Nov 03 16:44:10.750234 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:10.750184    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:44:15.310472 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:15.310431    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"]
Nov 03 16:44:15.310947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:15.310653    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy" containerID="cri-o://3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9" gracePeriod=30
Nov 03 16:44:19.871276 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:19.871243    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:44:20.632885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.632859    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:44:20.731719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.731693    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-certs\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.731884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.731740    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-token\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-token\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.731884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.731759    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-podinfo\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.732007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.731911    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-kube-api-access-k2hjb\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-kube-api-access-k2hjb\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.732007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.731973    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-socket\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.732111 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.731996    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-certs" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:44:20.732111 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732101    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istiod-ca-cert\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.732250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732130    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-envoy\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.732250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732154    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-data\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.732377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732250    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-socket" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:44:20.732377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732275    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-credential-socket\") pod \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\" (UID: \"93e24cbe-7c8d-4e26-8bd5-1252fe33594d\") "
Nov 03 16:44:20.732492 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732414    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-data" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:44:20.732756 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732729    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-credential-socket" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:44:20.732884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732805    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istiod-ca-cert" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:44:20.732884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732876    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.733015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732899    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.733015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732914    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.733015 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.732928    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.734232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.734179    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-kube-api-access-k2hjb" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "kube-api-access-k2hjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:44:20.734310 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.734251    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-token" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:44:20.734370 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.734317    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-envoy" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:44:20.734619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.734595    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-podinfo" pod "93e24cbe-7c8d-4e26-8bd5-1252fe33594d" (UID: "93e24cbe-7c8d-4e26-8bd5-1252fe33594d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:44:20.833840 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.833807    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hjb\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-kube-api-access-k2hjb\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.833840 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.833836    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.833840 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.833846    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.834147 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.833856    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.834147 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.833866    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e24cbe-7c8d-4e26-8bd5-1252fe33594d-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:44:20.957978 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.957942    5112 generic.go:356] "Generic (PLEG): container finished" podID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerID="3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9" exitCode=0
Nov 03 16:44:20.958445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.958010    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"
Nov 03 16:44:20.958445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.958031    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" event={"ID":"93e24cbe-7c8d-4e26-8bd5-1252fe33594d","Type":"ContainerDied","Data":"3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9"}
Nov 03 16:44:20.958445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.958088    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs" event={"ID":"93e24cbe-7c8d-4e26-8bd5-1252fe33594d","Type":"ContainerDied","Data":"7d50611f183ea31beaa3a70742b9c19a9df48807195871577bd0b701f01627d4"}
Nov 03 16:44:20.958445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.958117    5112 scope.go:122] "RemoveContainer" containerID="3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9"
Nov 03 16:44:20.971186 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.970814    5112 scope.go:122] "RemoveContainer" containerID="3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9"
Nov 03 16:44:20.971186 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:44:20.971145    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9\": container with ID starting with 3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9 not found: ID does not exist" containerID="3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9"
Nov 03 16:44:20.971394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:20.971176    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9"} err="failed to get container status \"3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9\": rpc error: code = NotFound desc = could not find container \"3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9\": container with ID starting with 3275b36595a81d097a08cd1f644bbfb4c48f5d57db187817d6576fa3578657a9 not found: ID does not exist"
Nov 03 16:44:21.133901 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:21.133868    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"]
Nov 03 16:44:21.143161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:21.143132    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-d5tfs"]
Nov 03 16:44:22.551184 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:22.551150    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" path="/var/lib/kubelet/pods/93e24cbe-7c8d-4e26-8bd5-1252fe33594d/volumes"
Nov 03 16:44:37.885463 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:44:37.885422    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:44:43.356193 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.356155    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"]
Nov 03 16:44:43.357333 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.357300    5112 cpu_manager.go:414] "containerMap: removing container" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy"
Nov 03 16:44:43.357333 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.357326    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy"
Nov 03 16:44:43.357568 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.357553    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e24cbe-7c8d-4e26-8bd5-1252fe33594d" containerName="istio-proxy"
Nov 03 16:44:43.363794 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.363770    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.417613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.417568    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"]
Nov 03 16:44:43.450471 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.450430    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-66f145-serving-cert\""
Nov 03 16:44:43.480242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.480191    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.480467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.480308    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcfh\" (UniqueName: \"kubernetes.io/projected/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-kube-api-access-bjcfh\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.480467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.480366    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-metrics-certs\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.480609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.480553    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-openshift-service-ca\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.582311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.582277    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-metrics-certs\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.582512 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.582413    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-openshift-service-ca\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.582582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.582533    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.582672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.582655    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcfh\" (UniqueName: \"kubernetes.io/projected/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-kube-api-access-bjcfh\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.582744 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:44:43.582690    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-serving-cert: secret "maas-api-e2e-api-b-66f145-serving-cert" not found
Nov 03 16:44:43.582799 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:44:43.582766    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls podName:53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6 nodeName:}" failed. No retries permitted until 2026-11-03 16:44:44.082745259 +0000 UTC m=+2045.662049108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls") pod "maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t" (UID: "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6") : secret "maas-api-e2e-api-b-66f145-serving-cert" not found
Nov 03 16:44:43.583237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.583192    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-openshift-service-ca\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.585490 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.585462    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-metrics-certs\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:43.671219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:43.656999    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcfh\" (UniqueName: \"kubernetes.io/projected/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-kube-api-access-bjcfh\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:44.096780 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:44.096688    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:44.099949 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:44.099922    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls\") pod \"maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:44.280052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:44.280009    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:44.661401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:44.661373    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"]
Nov 03 16:44:44.663365 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:44:44.663339    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f4793c_bc75_4bab_8ae9_40f1c7fa9ae6.slice/crio-08a7ae204ce5df6b81c1fcbaefd4e96b4f021a69e9a55bb81f9abebb30946c9d WatchSource:0}: Error finding container 08a7ae204ce5df6b81c1fcbaefd4e96b4f021a69e9a55bb81f9abebb30946c9d: Status 404 returned error can't find the container with id 08a7ae204ce5df6b81c1fcbaefd4e96b4f021a69e9a55bb81f9abebb30946c9d
Nov 03 16:44:45.186008 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:45.185969    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t" event={"ID":"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6","Type":"ContainerStarted","Data":"08a7ae204ce5df6b81c1fcbaefd4e96b4f021a69e9a55bb81f9abebb30946c9d"}
Nov 03 16:44:46.196658 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:46.196618    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t" event={"ID":"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6","Type":"ContainerStarted","Data":"bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b"}
Nov 03 16:44:46.197068 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:46.196745    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:44:46.314041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:46.313984    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t" podStartSLOduration=2.444152785 podStartE2EDuration="3.313970345s" totalImagesPullingTime="869.81756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:46.3114962 +0000 UTC m=+2047.890800046" watchObservedRunningTime="2026-11-03 16:44:46.313970345 +0000 UTC m=+2047.893274232"
Nov 03 16:44:52.211118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:44:52.211087    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:45:00.241741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.241703    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"]
Nov 03 16:45:00.248871 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.248836    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"]
Nov 03 16:45:00.249736 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.249704    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.254071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.254045    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"]
Nov 03 16:45:00.254223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.254121    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.259069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.259046    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.289262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.289238    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"]
Nov 03 16:45:00.293552 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.293534    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.320939 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.320910    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"]
Nov 03 16:45:00.333899 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.333864    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53e69f9-a100-441c-8314-0403e59067b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.333899 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.333906    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12df187a-0b04-4be7-b26d-2b9a82679574-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.334334 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.334313    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjfs\" (UniqueName: \"kubernetes.io/projected/ebab46a4-7079-4bdd-bb73-802f88ae096b-kube-api-access-6xjfs\") pod \"maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.334580 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.334555    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979x9\" (UniqueName: \"kubernetes.io/projected/e8732497-cd12-4507-b33e-e0d8cd74b0c4-kube-api-access-979x9\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.334646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.334616    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8sp\" (UniqueName: \"kubernetes.io/projected/12df187a-0b04-4be7-b26d-2b9a82679574-kube-api-access-pp8sp\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.334728 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.334645    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8732497-cd12-4507-b33e-e0d8cd74b0c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.334728 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.334675    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab46a4-7079-4bdd-bb73-802f88ae096b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.334728 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.334700    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cjzm\" (UniqueName: \"kubernetes.io/projected/a53e69f9-a100-441c-8314-0403e59067b7-kube-api-access-2cjzm\") pod \"maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.338881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.338851    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 16:45:00.361882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.360405    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"]
Nov 03 16:45:00.370483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.370411    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"]
Nov 03 16:45:00.393264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.393240    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"]
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.435866    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8sp\" (UniqueName: \"kubernetes.io/projected/12df187a-0b04-4be7-b26d-2b9a82679574-kube-api-access-pp8sp\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.435918    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8732497-cd12-4507-b33e-e0d8cd74b0c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.435951    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab46a4-7079-4bdd-bb73-802f88ae096b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.435976    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cjzm\" (UniqueName: \"kubernetes.io/projected/a53e69f9-a100-441c-8314-0403e59067b7-kube-api-access-2cjzm\") pod \"maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.436045    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53e69f9-a100-441c-8314-0403e59067b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.436077    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12df187a-0b04-4be7-b26d-2b9a82679574-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.436176    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjfs\" (UniqueName: \"kubernetes.io/projected/ebab46a4-7079-4bdd-bb73-802f88ae096b-kube-api-access-6xjfs\") pod \"maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.436476    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979x9\" (UniqueName: \"kubernetes.io/projected/e8732497-cd12-4507-b33e-e0d8cd74b0c4-kube-api-access-979x9\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.438251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.437769    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8732497-cd12-4507-b33e-e0d8cd74b0c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.441421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.440027    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab46a4-7079-4bdd-bb73-802f88ae096b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.441421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.440055    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53e69f9-a100-441c-8314-0403e59067b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.441421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.440689    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12df187a-0b04-4be7-b26d-2b9a82679574-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.495051 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.494893    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8sp\" (UniqueName: \"kubernetes.io/projected/12df187a-0b04-4be7-b26d-2b9a82679574-kube-api-access-pp8sp\") pod \"maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.495466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.495444    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-979x9\" (UniqueName: \"kubernetes.io/projected/e8732497-cd12-4507-b33e-e0d8cd74b0c4-kube-api-access-979x9\") pod \"maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.498827 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.498799    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cjzm\" (UniqueName: \"kubernetes.io/projected/a53e69f9-a100-441c-8314-0403e59067b7-kube-api-access-2cjzm\") pod \"maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.501396 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.501375    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjfs\" (UniqueName: \"kubernetes.io/projected/ebab46a4-7079-4bdd-bb73-802f88ae096b-kube-api-access-6xjfs\") pod \"maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.569131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.569098    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:00.576957 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.576927    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:00.582737 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.582707    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:00.615503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.615452    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:00.969690 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:00.969536    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"]
Nov 03 16:45:01.015436 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:01.015407    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"]
Nov 03 16:45:01.019620 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:01.019589    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"]
Nov 03 16:45:01.064565 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:01.064543    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"]
Nov 03 16:45:01.066600 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:45:01.066576    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12df187a_0b04_4be7_b26d_2b9a82679574.slice/crio-0c33cd9a2eb87eb172c42ed945897f5e2afc470f2de19eb7df146e1db56e2f7a WatchSource:0}: Error finding container 0c33cd9a2eb87eb172c42ed945897f5e2afc470f2de19eb7df146e1db56e2f7a: Status 404 returned error can't find the container with id 0c33cd9a2eb87eb172c42ed945897f5e2afc470f2de19eb7df146e1db56e2f7a
Nov 03 16:45:01.322308 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:01.322162    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" event={"ID":"e8732497-cd12-4507-b33e-e0d8cd74b0c4","Type":"ContainerStarted","Data":"974bf8a6d6deb0ae2d9722d980eba927fa50537030141f3df44b964f53737a29"}
Nov 03 16:45:01.323681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:01.323654    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" event={"ID":"ebab46a4-7079-4bdd-bb73-802f88ae096b","Type":"ContainerStarted","Data":"6a48516518067ed76d78dbb657ae2002278e26297fb59cc3df98fd483da7dce7"}
Nov 03 16:45:01.330633 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:01.325428    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" event={"ID":"a53e69f9-a100-441c-8314-0403e59067b7","Type":"ContainerStarted","Data":"412d095da2c85df8697456e427724baf22df8ac4043a1bf9f615be2f8726c3c3"}
Nov 03 16:45:01.338706 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:01.336064    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" event={"ID":"12df187a-0b04-4be7-b26d-2b9a82679574","Type":"ContainerStarted","Data":"0c33cd9a2eb87eb172c42ed945897f5e2afc470f2de19eb7df146e1db56e2f7a"}
Nov 03 16:45:02.345322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.345293    5112 generic.go:356] "Generic (PLEG): container finished" podID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerID="2a5f6a213c3d4c8b7a2a1854602c5c3b46801e1ad0254157a4007bf202927769" exitCode=60
Nov 03 16:45:02.345782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.345375    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" event={"ID":"ebab46a4-7079-4bdd-bb73-802f88ae096b","Type":"ContainerDied","Data":"2a5f6a213c3d4c8b7a2a1854602c5c3b46801e1ad0254157a4007bf202927769"}
Nov 03 16:45:02.345782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.345754    5112 scope.go:122] "RemoveContainer" containerID="2a5f6a213c3d4c8b7a2a1854602c5c3b46801e1ad0254157a4007bf202927769"
Nov 03 16:45:02.347264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.347240    5112 generic.go:356] "Generic (PLEG): container finished" podID="a53e69f9-a100-441c-8314-0403e59067b7" containerID="30c7e66d91b8e43f04a4b776cf833fee5be20ade967c7f2f05bf1220bfd314f9" exitCode=60
Nov 03 16:45:02.347393 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.347332    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" event={"ID":"a53e69f9-a100-441c-8314-0403e59067b7","Type":"ContainerDied","Data":"30c7e66d91b8e43f04a4b776cf833fee5be20ade967c7f2f05bf1220bfd314f9"}
Nov 03 16:45:02.347601 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.347583    5112 scope.go:122] "RemoveContainer" containerID="30c7e66d91b8e43f04a4b776cf833fee5be20ade967c7f2f05bf1220bfd314f9"
Nov 03 16:45:02.349164 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.349142    5112 generic.go:356] "Generic (PLEG): container finished" podID="12df187a-0b04-4be7-b26d-2b9a82679574" containerID="e8bc2ed6b7a79440854cbb849640e22cacdd66d4748be94c10c793be8a674062" exitCode=60
Nov 03 16:45:02.349345 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.349288    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" event={"ID":"12df187a-0b04-4be7-b26d-2b9a82679574","Type":"ContainerDied","Data":"e8bc2ed6b7a79440854cbb849640e22cacdd66d4748be94c10c793be8a674062"}
Nov 03 16:45:02.349532 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.349513    5112 scope.go:122] "RemoveContainer" containerID="e8bc2ed6b7a79440854cbb849640e22cacdd66d4748be94c10c793be8a674062"
Nov 03 16:45:02.351394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.351308    5112 generic.go:356] "Generic (PLEG): container finished" podID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerID="17fb2200abdfbca46b7d3c7c7e7dfaa4fe8566e4b247592a4d92b1290a414357" exitCode=60
Nov 03 16:45:02.351394 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.351342    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" event={"ID":"e8732497-cd12-4507-b33e-e0d8cd74b0c4","Type":"ContainerDied","Data":"17fb2200abdfbca46b7d3c7c7e7dfaa4fe8566e4b247592a4d92b1290a414357"}
Nov 03 16:45:02.351629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:02.351613    5112 scope.go:122] "RemoveContainer" containerID="17fb2200abdfbca46b7d3c7c7e7dfaa4fe8566e4b247592a4d92b1290a414357"
Nov 03 16:45:03.363731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.363691    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" event={"ID":"a53e69f9-a100-441c-8314-0403e59067b7","Type":"ContainerStarted","Data":"d58f1c9654ac2cf5c91ce0af8934eb545d74660148a6a91174198285d34b7344"}
Nov 03 16:45:03.366503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.366470    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" event={"ID":"12df187a-0b04-4be7-b26d-2b9a82679574","Type":"ContainerStarted","Data":"3d99ee61667a2d4679138489290fef4967f72467bf62f21286c7f887575f8eb5"}
Nov 03 16:45:03.369570 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.369539    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" event={"ID":"e8732497-cd12-4507-b33e-e0d8cd74b0c4","Type":"ContainerStarted","Data":"e57f794635b652798dae768522c75c6cac9e26671432a9b6e247596d57e50f8c"}
Nov 03 16:45:03.372619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.372593    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" event={"ID":"ebab46a4-7079-4bdd-bb73-802f88ae096b","Type":"ContainerStarted","Data":"e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92"}
Nov 03 16:45:03.372895 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.372880    5112 scope.go:122] "RemoveContainer" containerID="e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92"
Nov 03 16:45:03.373185 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:03.373157    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc_odh-ai-gateway-infra(ebab46a4-7079-4bdd-bb73-802f88ae096b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b"
Nov 03 16:45:03.482881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.482821    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" podStartSLOduration=3.4828042679999998 podStartE2EDuration="3.482804268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.479114546 +0000 UTC m=+2065.058418401" watchObservedRunningTime="2026-11-03 16:45:03.482804268 +0000 UTC m=+2065.062108125"
Nov 03 16:45:03.704822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.704768    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" podStartSLOduration=3.704753619 podStartE2EDuration="3.704753619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.700127721 +0000 UTC m=+2065.279431577" watchObservedRunningTime="2026-11-03 16:45:03.704753619 +0000 UTC m=+2065.284057533"
Nov 03 16:45:03.815371 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:03.815319    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" podStartSLOduration=3.815303589 podStartE2EDuration="3.815303589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.814141875 +0000 UTC m=+2065.393445753" watchObservedRunningTime="2026-11-03 16:45:03.815303589 +0000 UTC m=+2065.394607444"
Nov 03 16:45:04.388295 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.388249    5112 generic.go:356] "Generic (PLEG): container finished" podID="a53e69f9-a100-441c-8314-0403e59067b7" containerID="d58f1c9654ac2cf5c91ce0af8934eb545d74660148a6a91174198285d34b7344" exitCode=60
Nov 03 16:45:04.388813 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.388530    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" event={"ID":"a53e69f9-a100-441c-8314-0403e59067b7","Type":"ContainerDied","Data":"d58f1c9654ac2cf5c91ce0af8934eb545d74660148a6a91174198285d34b7344"}
Nov 03 16:45:04.388813 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.388644    5112 scope.go:122] "RemoveContainer" containerID="30c7e66d91b8e43f04a4b776cf833fee5be20ade967c7f2f05bf1220bfd314f9"
Nov 03 16:45:04.389189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.389165    5112 scope.go:122] "RemoveContainer" containerID="d58f1c9654ac2cf5c91ce0af8934eb545d74660148a6a91174198285d34b7344"
Nov 03 16:45:04.389886 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:04.389856    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df_odh-ai-gateway-infra(a53e69f9-a100-441c-8314-0403e59067b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" podUID="a53e69f9-a100-441c-8314-0403e59067b7"
Nov 03 16:45:04.392564 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.392533    5112 generic.go:356] "Generic (PLEG): container finished" podID="12df187a-0b04-4be7-b26d-2b9a82679574" containerID="3d99ee61667a2d4679138489290fef4967f72467bf62f21286c7f887575f8eb5" exitCode=60
Nov 03 16:45:04.392817 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.392794    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" event={"ID":"12df187a-0b04-4be7-b26d-2b9a82679574","Type":"ContainerDied","Data":"3d99ee61667a2d4679138489290fef4967f72467bf62f21286c7f887575f8eb5"}
Nov 03 16:45:04.393097 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.393073    5112 scope.go:122] "RemoveContainer" containerID="3d99ee61667a2d4679138489290fef4967f72467bf62f21286c7f887575f8eb5"
Nov 03 16:45:04.395879 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.395847    5112 generic.go:356] "Generic (PLEG): container finished" podID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerID="e57f794635b652798dae768522c75c6cac9e26671432a9b6e247596d57e50f8c" exitCode=60
Nov 03 16:45:04.396009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.395982    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" event={"ID":"e8732497-cd12-4507-b33e-e0d8cd74b0c4","Type":"ContainerDied","Data":"e57f794635b652798dae768522c75c6cac9e26671432a9b6e247596d57e50f8c"}
Nov 03 16:45:04.396497 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.396471    5112 scope.go:122] "RemoveContainer" containerID="e57f794635b652798dae768522c75c6cac9e26671432a9b6e247596d57e50f8c"
Nov 03 16:45:04.396947 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:04.396918    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9_odh-ai-gateway-infra(e8732497-cd12-4507-b33e-e0d8cd74b0c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4"
Nov 03 16:45:04.398638 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.398610    5112 generic.go:356] "Generic (PLEG): container finished" podID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerID="e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92" exitCode=60
Nov 03 16:45:04.399640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.398763    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" event={"ID":"ebab46a4-7079-4bdd-bb73-802f88ae096b","Type":"ContainerDied","Data":"e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92"}
Nov 03 16:45:04.399640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.399180    5112 scope.go:122] "RemoveContainer" containerID="e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92"
Nov 03 16:45:04.399780 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:04.399633    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc_odh-ai-gateway-infra(ebab46a4-7079-4bdd-bb73-802f88ae096b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b"
Nov 03 16:45:04.402108 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:04.401862    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx_odh-ai-gateway-infra(12df187a-0b04-4be7-b26d-2b9a82679574)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" podUID="12df187a-0b04-4be7-b26d-2b9a82679574"
Nov 03 16:45:04.410773 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.410749    5112 scope.go:122] "RemoveContainer" containerID="e8bc2ed6b7a79440854cbb849640e22cacdd66d4748be94c10c793be8a674062"
Nov 03 16:45:04.427084 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.427062    5112 scope.go:122] "RemoveContainer" containerID="17fb2200abdfbca46b7d3c7c7e7dfaa4fe8566e4b247592a4d92b1290a414357"
Nov 03 16:45:04.441942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:04.441924    5112 scope.go:122] "RemoveContainer" containerID="2a5f6a213c3d4c8b7a2a1854602c5c3b46801e1ad0254157a4007bf202927769"
Nov 03 16:45:05.409140 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:05.409113    5112 scope.go:122] "RemoveContainer" containerID="e57f794635b652798dae768522c75c6cac9e26671432a9b6e247596d57e50f8c"
Nov 03 16:45:05.409672 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:05.409528    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9_odh-ai-gateway-infra(e8732497-cd12-4507-b33e-e0d8cd74b0c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4"
Nov 03 16:45:05.410907 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:05.410884    5112 scope.go:122] "RemoveContainer" containerID="e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92"
Nov 03 16:45:05.411302 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:05.411278    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc_odh-ai-gateway-infra(ebab46a4-7079-4bdd-bb73-802f88ae096b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b"
Nov 03 16:45:05.412349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:05.412330    5112 scope.go:122] "RemoveContainer" containerID="d58f1c9654ac2cf5c91ce0af8934eb545d74660148a6a91174198285d34b7344"
Nov 03 16:45:05.412626 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:05.412595    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df_odh-ai-gateway-infra(a53e69f9-a100-441c-8314-0403e59067b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" podUID="a53e69f9-a100-441c-8314-0403e59067b7"
Nov 03 16:45:05.414028 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:05.414010    5112 scope.go:122] "RemoveContainer" containerID="3d99ee61667a2d4679138489290fef4967f72467bf62f21286c7f887575f8eb5"
Nov 03 16:45:05.414257 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:05.414236    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx_odh-ai-gateway-infra(12df187a-0b04-4be7-b26d-2b9a82679574)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" podUID="12df187a-0b04-4be7-b26d-2b9a82679574"
Nov 03 16:45:07.886227 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:07.886176    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:45:11.673817 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.673778    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"]
Nov 03 16:45:11.680036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.680015    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.732322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.732294    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 16:45:11.785502 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.785464    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"]
Nov 03 16:45:11.806060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.806005    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4chm\" (UniqueName: \"kubernetes.io/projected/c094c8be-971f-42ce-b2b2-5b05f36a94a6-kube-api-access-v4chm\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.806311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.806290    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.806391 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.806352    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.908190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.908151    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.908430 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.908290    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.908430 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.908402    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4chm\" (UniqueName: \"kubernetes.io/projected/c094c8be-971f-42ce-b2b2-5b05f36a94a6-kube-api-access-v4chm\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.909054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.909033    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.910768 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.910741    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.966273 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.966154    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4chm\" (UniqueName: \"kubernetes.io/projected/c094c8be-971f-42ce-b2b2-5b05f36a94a6-kube-api-access-v4chm\") pod \"maas-api-revoke-sub-52a433e7-v6xkw\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:11.995078 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:11.995041    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:12.386100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:12.386064    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"]
Nov 03 16:45:12.387730 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:45:12.387694    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc094c8be_971f_42ce_b2b2_5b05f36a94a6.slice/crio-8e82eff9b542035b36482b2fba699ba14697f362b18fa845583638c6ce4280ef WatchSource:0}: Error finding container 8e82eff9b542035b36482b2fba699ba14697f362b18fa845583638c6ce4280ef: Status 404 returned error can't find the container with id 8e82eff9b542035b36482b2fba699ba14697f362b18fa845583638c6ce4280ef
Nov 03 16:45:12.509532 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:12.509419    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw" event={"ID":"c094c8be-971f-42ce-b2b2-5b05f36a94a6","Type":"ContainerStarted","Data":"8e82eff9b542035b36482b2fba699ba14697f362b18fa845583638c6ce4280ef"}
Nov 03 16:45:13.520060 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:13.519968    5112 generic.go:356] "Generic (PLEG): container finished" podID="c094c8be-971f-42ce-b2b2-5b05f36a94a6" containerID="95877921aa1ad7a2d3d3174a1b9822a2b84608fdd5cd8dc68483998ed6fce3f7" exitCode=0
Nov 03 16:45:13.520515 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:13.520054    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw" event={"ID":"c094c8be-971f-42ce-b2b2-5b05f36a94a6","Type":"ContainerDied","Data":"95877921aa1ad7a2d3d3174a1b9822a2b84608fdd5cd8dc68483998ed6fce3f7"}
Nov 03 16:45:14.723009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.722983    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:14.863727 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.863643    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-openshift-ca\") pod \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") "
Nov 03 16:45:14.863727 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.863688    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-xks-ca\") pod \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") "
Nov 03 16:45:14.863950 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.863800    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c094c8be-971f-42ce-b2b2-5b05f36a94a6-kube-api-access-v4chm\" (UniqueName: \"kubernetes.io/projected/c094c8be-971f-42ce-b2b2-5b05f36a94a6-kube-api-access-v4chm\") pod \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\" (UID: \"c094c8be-971f-42ce-b2b2-5b05f36a94a6\") "
Nov 03 16:45:14.864451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.864415    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-openshift-ca" pod "c094c8be-971f-42ce-b2b2-5b05f36a94a6" (UID: "c094c8be-971f-42ce-b2b2-5b05f36a94a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:45:14.864632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.864605    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:14.867062 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.867034    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c094c8be-971f-42ce-b2b2-5b05f36a94a6-kube-api-access-v4chm" pod "c094c8be-971f-42ce-b2b2-5b05f36a94a6" (UID: "c094c8be-971f-42ce-b2b2-5b05f36a94a6"). InnerVolumeSpecName "kube-api-access-v4chm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:45:14.871477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.871452    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-xks-ca" pod "c094c8be-971f-42ce-b2b2-5b05f36a94a6" (UID: "c094c8be-971f-42ce-b2b2-5b05f36a94a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:45:14.975566 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.975522    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c094c8be-971f-42ce-b2b2-5b05f36a94a6-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:14.975566 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:14.975566    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4chm\" (UniqueName: \"kubernetes.io/projected/c094c8be-971f-42ce-b2b2-5b05f36a94a6-kube-api-access-v4chm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:15.539168 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:15.539141    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"
Nov 03 16:45:15.539168 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:15.539166    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw" event={"ID":"c094c8be-971f-42ce-b2b2-5b05f36a94a6","Type":"ContainerDied","Data":"8e82eff9b542035b36482b2fba699ba14697f362b18fa845583638c6ce4280ef"}
Nov 03 16:45:15.539397 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:15.539240    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e82eff9b542035b36482b2fba699ba14697f362b18fa845583638c6ce4280ef"
Nov 03 16:45:15.542282 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:15.542261    5112 scope.go:122] "RemoveContainer" containerID="3d99ee61667a2d4679138489290fef4967f72467bf62f21286c7f887575f8eb5"
Nov 03 16:45:15.542534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:15.542518    5112 scope.go:122] "RemoveContainer" containerID="e57f794635b652798dae768522c75c6cac9e26671432a9b6e247596d57e50f8c"
Nov 03 16:45:16.543031 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.543002    5112 scope.go:122] "RemoveContainer" containerID="e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92"
Nov 03 16:45:16.550113 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.550086    5112 generic.go:356] "Generic (PLEG): container finished" podID="12df187a-0b04-4be7-b26d-2b9a82679574" containerID="545e49f6f954587b94dc10060e5ff0460b11c27a3073002823d3fabbfba38338" exitCode=60
Nov 03 16:45:16.552052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.552024    5112 generic.go:356] "Generic (PLEG): container finished" podID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerID="6181469f1954a3639f5681e12d37a6bd40d9289580fb8a1b8c8804749600edfd" exitCode=60
Nov 03 16:45:16.553313 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.553287    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" event={"ID":"12df187a-0b04-4be7-b26d-2b9a82679574","Type":"ContainerDied","Data":"545e49f6f954587b94dc10060e5ff0460b11c27a3073002823d3fabbfba38338"}
Nov 03 16:45:16.553381 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.553348    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" event={"ID":"e8732497-cd12-4507-b33e-e0d8cd74b0c4","Type":"ContainerDied","Data":"6181469f1954a3639f5681e12d37a6bd40d9289580fb8a1b8c8804749600edfd"}
Nov 03 16:45:16.553431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.553413    5112 scope.go:122] "RemoveContainer" containerID="3d99ee61667a2d4679138489290fef4967f72467bf62f21286c7f887575f8eb5"
Nov 03 16:45:16.553661 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.553631    5112 scope.go:122] "RemoveContainer" containerID="545e49f6f954587b94dc10060e5ff0460b11c27a3073002823d3fabbfba38338"
Nov 03 16:45:16.553872 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:16.553849    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx_odh-ai-gateway-infra(12df187a-0b04-4be7-b26d-2b9a82679574)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" podUID="12df187a-0b04-4be7-b26d-2b9a82679574"
Nov 03 16:45:16.553951 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.553897    5112 scope.go:122] "RemoveContainer" containerID="6181469f1954a3639f5681e12d37a6bd40d9289580fb8a1b8c8804749600edfd"
Nov 03 16:45:16.554161 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:16.554133    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9_odh-ai-gateway-infra(e8732497-cd12-4507-b33e-e0d8cd74b0c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4"
Nov 03 16:45:16.598078 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:16.598063    5112 scope.go:122] "RemoveContainer" containerID="e57f794635b652798dae768522c75c6cac9e26671432a9b6e247596d57e50f8c"
Nov 03 16:45:17.543000 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.542965    5112 scope.go:122] "RemoveContainer" containerID="d58f1c9654ac2cf5c91ce0af8934eb545d74660148a6a91174198285d34b7344"
Nov 03 16:45:17.565645 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.565619    5112 generic.go:356] "Generic (PLEG): container finished" podID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerID="e5e94909435eca247c7c8b7df3992131882f3b0809503511754e8817c7d0bf53" exitCode=60
Nov 03 16:45:17.566052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.565684    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" event={"ID":"ebab46a4-7079-4bdd-bb73-802f88ae096b","Type":"ContainerDied","Data":"e5e94909435eca247c7c8b7df3992131882f3b0809503511754e8817c7d0bf53"}
Nov 03 16:45:17.566052 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.565741    5112 scope.go:122] "RemoveContainer" containerID="e43b1f0afe45f741993721bb46f9647bcee1096de1b495100302ab2512c50f92"
Nov 03 16:45:17.566162 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.566131    5112 scope.go:122] "RemoveContainer" containerID="e5e94909435eca247c7c8b7df3992131882f3b0809503511754e8817c7d0bf53"
Nov 03 16:45:17.566527 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:17.566460    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc_odh-ai-gateway-infra(ebab46a4-7079-4bdd-bb73-802f88ae096b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b"
Nov 03 16:45:17.686245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.686193    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"]
Nov 03 16:45:17.794858 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.794816    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"]
Nov 03 16:45:17.890769 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.890706    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:17.925384 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.923656    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12df187a-0b04-4be7-b26d-2b9a82679574-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12df187a-0b04-4be7-b26d-2b9a82679574-maas-api-service-ca\") pod \"12df187a-0b04-4be7-b26d-2b9a82679574\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") "
Nov 03 16:45:17.925384 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.923870    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12df187a-0b04-4be7-b26d-2b9a82679574-kube-api-access-pp8sp\" (UniqueName: \"kubernetes.io/projected/12df187a-0b04-4be7-b26d-2b9a82679574-kube-api-access-pp8sp\") pod \"12df187a-0b04-4be7-b26d-2b9a82679574\" (UID: \"12df187a-0b04-4be7-b26d-2b9a82679574\") "
Nov 03 16:45:17.925384 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.924082    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12df187a-0b04-4be7-b26d-2b9a82679574-maas-api-service-ca" pod "12df187a-0b04-4be7-b26d-2b9a82679574" (UID: "12df187a-0b04-4be7-b26d-2b9a82679574"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:45:17.925384 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.924558    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12df187a-0b04-4be7-b26d-2b9a82679574-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:17.927443 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.927410    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12df187a-0b04-4be7-b26d-2b9a82679574-kube-api-access-pp8sp" pod "12df187a-0b04-4be7-b26d-2b9a82679574" (UID: "12df187a-0b04-4be7-b26d-2b9a82679574"). InnerVolumeSpecName "kube-api-access-pp8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:45:17.970916 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:17.970890    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:18.025859 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.025776    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8732497-cd12-4507-b33e-e0d8cd74b0c4-kube-api-access-979x9\" (UniqueName: \"kubernetes.io/projected/e8732497-cd12-4507-b33e-e0d8cd74b0c4-kube-api-access-979x9\") pod \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") "
Nov 03 16:45:18.026040 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.025881    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8732497-cd12-4507-b33e-e0d8cd74b0c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8732497-cd12-4507-b33e-e0d8cd74b0c4-maas-api-service-ca\") pod \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\" (UID: \"e8732497-cd12-4507-b33e-e0d8cd74b0c4\") "
Nov 03 16:45:18.026301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.026276    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8732497-cd12-4507-b33e-e0d8cd74b0c4-maas-api-service-ca" pod "e8732497-cd12-4507-b33e-e0d8cd74b0c4" (UID: "e8732497-cd12-4507-b33e-e0d8cd74b0c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:45:18.026484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.026466    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8sp\" (UniqueName: \"kubernetes.io/projected/12df187a-0b04-4be7-b26d-2b9a82679574-kube-api-access-pp8sp\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:18.026529 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.026490    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8732497-cd12-4507-b33e-e0d8cd74b0c4-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:18.028512 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.028483    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8732497-cd12-4507-b33e-e0d8cd74b0c4-kube-api-access-979x9" pod "e8732497-cd12-4507-b33e-e0d8cd74b0c4" (UID: "e8732497-cd12-4507-b33e-e0d8cd74b0c4"). InnerVolumeSpecName "kube-api-access-979x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:45:18.128421 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.128385    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979x9\" (UniqueName: \"kubernetes.io/projected/e8732497-cd12-4507-b33e-e0d8cd74b0c4-kube-api-access-979x9\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:18.576982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.576948    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" event={"ID":"a53e69f9-a100-441c-8314-0403e59067b7","Type":"ContainerStarted","Data":"1f04e02e642e142138cb967f5d624647b00d1bc808d0c51fba90448ab2570326"}
Nov 03 16:45:18.578489 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.578464    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"
Nov 03 16:45:18.578622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.578490    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx" event={"ID":"12df187a-0b04-4be7-b26d-2b9a82679574","Type":"ContainerDied","Data":"0c33cd9a2eb87eb172c42ed945897f5e2afc470f2de19eb7df146e1db56e2f7a"}
Nov 03 16:45:18.578622 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.578552    5112 scope.go:122] "RemoveContainer" containerID="545e49f6f954587b94dc10060e5ff0460b11c27a3073002823d3fabbfba38338"
Nov 03 16:45:18.580066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.580042    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"
Nov 03 16:45:18.580235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.580073    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9" event={"ID":"e8732497-cd12-4507-b33e-e0d8cd74b0c4","Type":"ContainerDied","Data":"974bf8a6d6deb0ae2d9722d980eba927fa50537030141f3df44b964f53737a29"}
Nov 03 16:45:18.591260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.591237    5112 scope.go:122] "RemoveContainer" containerID="6181469f1954a3639f5681e12d37a6bd40d9289580fb8a1b8c8804749600edfd"
Nov 03 16:45:18.708759 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.708727    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"]
Nov 03 16:45:18.812887 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.812852    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"]
Nov 03 16:45:18.824687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.824654    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6ae690-29895405-7h6d9"]
Nov 03 16:45:18.893646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.893614    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:18.935948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.935909    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"]
Nov 03 16:45:18.942671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.942637    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2bb620-29895405-2xpbx"]
Nov 03 16:45:18.942850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.942733    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebab46a4-7079-4bdd-bb73-802f88ae096b-kube-api-access-6xjfs\" (UniqueName: \"kubernetes.io/projected/ebab46a4-7079-4bdd-bb73-802f88ae096b-kube-api-access-6xjfs\") pod \"ebab46a4-7079-4bdd-bb73-802f88ae096b\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") "
Nov 03 16:45:18.943010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.942984    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebab46a4-7079-4bdd-bb73-802f88ae096b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab46a4-7079-4bdd-bb73-802f88ae096b-maas-api-service-ca\") pod \"ebab46a4-7079-4bdd-bb73-802f88ae096b\" (UID: \"ebab46a4-7079-4bdd-bb73-802f88ae096b\") "
Nov 03 16:45:18.943545 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.943517    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebab46a4-7079-4bdd-bb73-802f88ae096b-maas-api-service-ca" pod "ebab46a4-7079-4bdd-bb73-802f88ae096b" (UID: "ebab46a4-7079-4bdd-bb73-802f88ae096b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:45:18.945673 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:18.945643    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebab46a4-7079-4bdd-bb73-802f88ae096b-kube-api-access-6xjfs" pod "ebab46a4-7079-4bdd-bb73-802f88ae096b" (UID: "ebab46a4-7079-4bdd-bb73-802f88ae096b"). InnerVolumeSpecName "kube-api-access-6xjfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:45:19.045686 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.045619    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab46a4-7079-4bdd-bb73-802f88ae096b-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:19.045686 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.045647    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjfs\" (UniqueName: \"kubernetes.io/projected/ebab46a4-7079-4bdd-bb73-802f88ae096b-kube-api-access-6xjfs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:19.593229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.593171    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"
Nov 03 16:45:19.593730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.593172    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc" event={"ID":"ebab46a4-7079-4bdd-bb73-802f88ae096b","Type":"ContainerDied","Data":"6a48516518067ed76d78dbb657ae2002278e26297fb59cc3df98fd483da7dce7"}
Nov 03 16:45:19.593730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.593312    5112 scope.go:122] "RemoveContainer" containerID="e5e94909435eca247c7c8b7df3992131882f3b0809503511754e8817c7d0bf53"
Nov 03 16:45:19.595508 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.595459    5112 generic.go:356] "Generic (PLEG): container finished" podID="a53e69f9-a100-441c-8314-0403e59067b7" containerID="1f04e02e642e142138cb967f5d624647b00d1bc808d0c51fba90448ab2570326" exitCode=60
Nov 03 16:45:19.595613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.595516    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" event={"ID":"a53e69f9-a100-441c-8314-0403e59067b7","Type":"ContainerDied","Data":"1f04e02e642e142138cb967f5d624647b00d1bc808d0c51fba90448ab2570326"}
Nov 03 16:45:19.595962 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.595943    5112 scope.go:122] "RemoveContainer" containerID="1f04e02e642e142138cb967f5d624647b00d1bc808d0c51fba90448ab2570326"
Nov 03 16:45:19.596354 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:19.596331    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df_odh-ai-gateway-infra(a53e69f9-a100-441c-8314-0403e59067b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" podUID="a53e69f9-a100-441c-8314-0403e59067b7"
Nov 03 16:45:19.604758 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.604733    5112 scope.go:122] "RemoveContainer" containerID="d58f1c9654ac2cf5c91ce0af8934eb545d74660148a6a91174198285d34b7344"
Nov 03 16:45:19.714636 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.714597    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"]
Nov 03 16:45:19.757371 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.757335    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"]
Nov 03 16:45:19.763444 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:19.763416    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-23c440-29895405-wn4rc"]
Nov 03 16:45:20.554399 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.554364    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" path="/var/lib/kubelet/pods/12df187a-0b04-4be7-b26d-2b9a82679574/volumes"
Nov 03 16:45:20.555045 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.555023    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" path="/var/lib/kubelet/pods/e8732497-cd12-4507-b33e-e0d8cd74b0c4/volumes"
Nov 03 16:45:20.555556 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.555533    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" path="/var/lib/kubelet/pods/ebab46a4-7079-4bdd-bb73-802f88ae096b/volumes"
Nov 03 16:45:20.814938 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.814870    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:20.870610 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.870576    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53e69f9-a100-441c-8314-0403e59067b7-kube-api-access-2cjzm\" (UniqueName: \"kubernetes.io/projected/a53e69f9-a100-441c-8314-0403e59067b7-kube-api-access-2cjzm\") pod \"a53e69f9-a100-441c-8314-0403e59067b7\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") "
Nov 03 16:45:20.870881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.870844    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53e69f9-a100-441c-8314-0403e59067b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53e69f9-a100-441c-8314-0403e59067b7-maas-api-service-ca\") pod \"a53e69f9-a100-441c-8314-0403e59067b7\" (UID: \"a53e69f9-a100-441c-8314-0403e59067b7\") "
Nov 03 16:45:20.872169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.871669    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53e69f9-a100-441c-8314-0403e59067b7-maas-api-service-ca" pod "a53e69f9-a100-441c-8314-0403e59067b7" (UID: "a53e69f9-a100-441c-8314-0403e59067b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:45:20.872169 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.872134    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53e69f9-a100-441c-8314-0403e59067b7-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:20.873091 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.873065    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53e69f9-a100-441c-8314-0403e59067b7-kube-api-access-2cjzm" pod "a53e69f9-a100-441c-8314-0403e59067b7" (UID: "a53e69f9-a100-441c-8314-0403e59067b7"). InnerVolumeSpecName "kube-api-access-2cjzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:45:20.974265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:20.974222    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cjzm\" (UniqueName: \"kubernetes.io/projected/a53e69f9-a100-441c-8314-0403e59067b7-kube-api-access-2cjzm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:45:21.624539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:21.624506    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"
Nov 03 16:45:21.624749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:21.624506    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df" event={"ID":"a53e69f9-a100-441c-8314-0403e59067b7","Type":"ContainerDied","Data":"412d095da2c85df8697456e427724baf22df8ac4043a1bf9f615be2f8726c3c3"}
Nov 03 16:45:21.624749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:21.624662    5112 scope.go:122] "RemoveContainer" containerID="1f04e02e642e142138cb967f5d624647b00d1bc808d0c51fba90448ab2570326"
Nov 03 16:45:21.800322 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:21.799510    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"]
Nov 03 16:45:21.837827 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:21.837788    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0fda73-29895405-ln8df"]
Nov 03 16:45:22.638244 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:22.620947    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53e69f9-a100-441c-8314-0403e59067b7" path="/var/lib/kubelet/pods/a53e69f9-a100-441c-8314-0403e59067b7/volumes"
Nov 03 16:45:23.195792 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.195755    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"]
Nov 03 16:45:23.196909 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.196886    5112 cpu_manager.go:414] "containerMap: removing container" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.196961 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.196912    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.196961 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.196928    5112 cpu_manager.go:414] "containerMap: removing container" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:23.196961 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.196936    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.196966    5112 cpu_manager.go:414] "containerMap: removing container" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.196973    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.196997    5112 cpu_manager.go:414] "containerMap: removing container" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197004    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197019    5112 cpu_manager.go:414] "containerMap: removing container" podUID="c094c8be-971f-42ce-b2b2-5b05f36a94a6" containerName="revoke-keys"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197026    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c094c8be-971f-42ce-b2b2-5b05f36a94a6" containerName="revoke-keys"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197054    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:23.197066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197061    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197079    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197086    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197104    5112 cpu_manager.go:414] "containerMap: removing container" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197111    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197131    5112 cpu_manager.go:414] "containerMap: removing container" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197138    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197156    5112 cpu_manager.go:414] "containerMap: removing container" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:23.197321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197163    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197351    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197368    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197379    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197389    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197401    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197412    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197420    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8732497-cd12-4507-b33e-e0d8cd74b0c4" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197430    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197444    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="c094c8be-971f-42ce-b2b2-5b05f36a94a6" containerName="revoke-keys"
Nov 03 16:45:23.197740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.197459    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:23.203293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.203270    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.255788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.255760    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"]
Nov 03 16:45:23.290872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.290834    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00cbb13-70ee-40a0-a1f8-f4982b375909-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.291054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.290947    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-metrics-certs\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.291054 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.290994    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.291175 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.291069    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbn7t\" (UniqueName: \"kubernetes.io/projected/a00cbb13-70ee-40a0-a1f8-f4982b375909-kube-api-access-dbn7t\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.296542 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.296443    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0ea8a5-serving-cert\""
Nov 03 16:45:23.392273 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.392240    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00cbb13-70ee-40a0-a1f8-f4982b375909-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.392479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.392331    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-metrics-certs\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.392479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.392361    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.392479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.392413    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbn7t\" (UniqueName: \"kubernetes.io/projected/a00cbb13-70ee-40a0-a1f8-f4982b375909-kube-api-access-dbn7t\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.392769 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:23.392739    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-serving-cert: secret "maas-api-e2e-inf-b-0ea8a5-serving-cert" not found
Nov 03 16:45:23.392922 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:23.392798    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls podName:a00cbb13-70ee-40a0-a1f8-f4982b375909 nodeName:}" failed. No retries permitted until 2026-11-03 16:45:23.892783988 +0000 UTC m=+2085.472087835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls") pod "maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25" (UID: "a00cbb13-70ee-40a0-a1f8-f4982b375909") : secret "maas-api-e2e-inf-b-0ea8a5-serving-cert" not found
Nov 03 16:45:23.393087 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.393062    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00cbb13-70ee-40a0-a1f8-f4982b375909-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.395121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.395097    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-metrics-certs\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.513144 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.513050    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbn7t\" (UniqueName: \"kubernetes.io/projected/a00cbb13-70ee-40a0-a1f8-f4982b375909-kube-api-access-dbn7t\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.906167 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.906132    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:23.908720 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:23.908689    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls\") pod \"maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:24.120065 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:24.120020    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:24.513104 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:24.512868    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"]
Nov 03 16:45:24.706260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:24.706213    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25" event={"ID":"a00cbb13-70ee-40a0-a1f8-f4982b375909","Type":"ContainerStarted","Data":"e6f547f5399ccfade6b11b7e752e1786264861eef755a7a4e6f2d6690d53e1e1"}
Nov 03 16:45:25.715994 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:25.715964    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25" event={"ID":"a00cbb13-70ee-40a0-a1f8-f4982b375909","Type":"ContainerStarted","Data":"63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e"}
Nov 03 16:45:25.716418 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:25.716028    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:25.832367 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:25.832252    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25" podStartSLOduration=1.848381826 podStartE2EDuration="2.832236805s" totalImagesPullingTime="983.854979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:25.829106318 +0000 UTC m=+2087.408410172" watchObservedRunningTime="2026-11-03 16:45:25.832236805 +0000 UTC m=+2087.411540660"
Nov 03 16:45:31.730767 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:31.730729    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:45:33.724303 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.724259    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"]
Nov 03 16:45:33.726687 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.726656    5112 cpu_manager.go:414] "containerMap: removing container" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:33.726866 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.726850    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:33.727036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.727024    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:33.727119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.727108    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:33.727281 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.727269    5112 cpu_manager.go:414] "containerMap: removing container" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:33.727377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.727366    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:33.727798 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.727779    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebab46a4-7079-4bdd-bb73-802f88ae096b" containerName="cleanup"
Nov 03 16:45:33.727950 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.727939    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="12df187a-0b04-4be7-b26d-2b9a82679574" containerName="cleanup"
Nov 03 16:45:33.728040 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.728030    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53e69f9-a100-441c-8314-0403e59067b7" containerName="cleanup"
Nov 03 16:45:33.734362 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.734341    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:33.783005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.782968    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"]
Nov 03 16:45:33.824247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.823403    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6822cb-serving-cert\""
Nov 03 16:45:33.904303 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.904260    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4c9r\" (UniqueName: \"kubernetes.io/projected/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-kube-api-access-j4c9r\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:33.904501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.904341    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:33.904501 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.904456    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:33.904630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:33.904582    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-metrics-certs\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.010671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.005993    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-metrics-certs\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.010671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.008581    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c9r\" (UniqueName: \"kubernetes.io/projected/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-kube-api-access-j4c9r\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.010671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.008645    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.010671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.008763    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.010671 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:34.009021    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-serving-cert: secret "maas-api-e2e-inf-a-6822cb-serving-cert" not found
Nov 03 16:45:34.010671 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:34.009184    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls podName:6835a522-d79e-4a1c-8dbf-8a6d04dc28e7 nodeName:}" failed. No retries permitted until 2026-11-03 16:45:34.509161642 +0000 UTC m=+2096.088465494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls") pod "maas-api-e2e-inf-a-6822cb-74f98947-p9v5v" (UID: "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7") : secret "maas-api-e2e-inf-a-6822cb-serving-cert" not found
Nov 03 16:45:34.010671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.010512    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.190226 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.189860    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-metrics-certs\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.191229 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.191181    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4c9r\" (UniqueName: \"kubernetes.io/projected/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-kube-api-access-j4c9r\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.521236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.521175    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.523799 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.523769    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6822cb-74f98947-p9v5v\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:34.653643 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:34.653613    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:35.044951 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:35.044915    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"]
Nov 03 16:45:35.045327 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:45:35.045085    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6835a522_d79e_4a1c_8dbf_8a6d04dc28e7.slice/crio-52c50840047a59d2bdc6b4d6f72254692104e5a92d842cc4680b9ff8fa93da0b WatchSource:0}: Error finding container 52c50840047a59d2bdc6b4d6f72254692104e5a92d842cc4680b9ff8fa93da0b: Status 404 returned error can't find the container with id 52c50840047a59d2bdc6b4d6f72254692104e5a92d842cc4680b9ff8fa93da0b
Nov 03 16:45:35.848816 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:35.848779    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v" event={"ID":"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7","Type":"ContainerStarted","Data":"52c50840047a59d2bdc6b4d6f72254692104e5a92d842cc4680b9ff8fa93da0b"}
Nov 03 16:45:36.859160 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:36.859120    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v" event={"ID":"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7","Type":"ContainerStarted","Data":"c976585d3cb66107f0b949e702725d6d694726749de3b1781a913d362dadde7d"}
Nov 03 16:45:36.859653 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:36.859421    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:45:37.012223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:37.010701    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v" podStartSLOduration=3.193969285 podStartE2EDuration="4.010676568s" totalImagesPullingTime="816.707283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:36.972711329 +0000 UTC m=+2098.552015209" watchObservedRunningTime="2026-11-03 16:45:37.010676568 +0000 UTC m=+2098.589980424"
Nov 03 16:45:37.886154 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:45:37.886115    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:45:38.925102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:38.925065    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:45:38.928548 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:38.928520    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:45:39.010220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:39.008145    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:45:39.010220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:39.008954    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:45:42.875313 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:45:42.875276    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:46:07.886456 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:46:07.886415    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:46:37.891661 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:46:37.891615    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:47:00.028057 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.027964    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"]
Nov 03 16:47:00.028703 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.028182    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9" podUID="07a6ee17-7736-44a8-8a55-9ad954129b8b" containerName="maas-api" containerID="cri-o://4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342" gracePeriod=30
Nov 03 16:47:00.384279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.384251    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:47:00.509219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.505431    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a6ee17-7736-44a8-8a55-9ad954129b8b-kube-api-access-2wf5p\" (UniqueName: \"kubernetes.io/projected/07a6ee17-7736-44a8-8a55-9ad954129b8b-kube-api-access-2wf5p\") pod \"07a6ee17-7736-44a8-8a55-9ad954129b8b\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") "
Nov 03 16:47:00.509219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.505727    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a6ee17-7736-44a8-8a55-9ad954129b8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6ee17-7736-44a8-8a55-9ad954129b8b-openshift-service-ca\") pod \"07a6ee17-7736-44a8-8a55-9ad954129b8b\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") "
Nov 03 16:47:00.509219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.505761    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-metrics-certs\") pod \"07a6ee17-7736-44a8-8a55-9ad954129b8b\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") "
Nov 03 16:47:00.509219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.505832    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls\") pod \"07a6ee17-7736-44a8-8a55-9ad954129b8b\" (UID: \"07a6ee17-7736-44a8-8a55-9ad954129b8b\") "
Nov 03 16:47:00.511962 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.511650    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a6ee17-7736-44a8-8a55-9ad954129b8b-openshift-service-ca" pod "07a6ee17-7736-44a8-8a55-9ad954129b8b" (UID: "07a6ee17-7736-44a8-8a55-9ad954129b8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:47:00.517642 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.517264    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a6ee17-7736-44a8-8a55-9ad954129b8b-kube-api-access-2wf5p" pod "07a6ee17-7736-44a8-8a55-9ad954129b8b" (UID: "07a6ee17-7736-44a8-8a55-9ad954129b8b"). InnerVolumeSpecName "kube-api-access-2wf5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:47:00.522161 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.522133    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls" pod "07a6ee17-7736-44a8-8a55-9ad954129b8b" (UID: "07a6ee17-7736-44a8-8a55-9ad954129b8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:47:00.531356 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.531324    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-metrics-certs" pod "07a6ee17-7736-44a8-8a55-9ad954129b8b" (UID: "07a6ee17-7736-44a8-8a55-9ad954129b8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:47:00.608918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.608838    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6ee17-7736-44a8-8a55-9ad954129b8b-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:00.608918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.608878    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:00.608918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.608896    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a6ee17-7736-44a8-8a55-9ad954129b8b-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:00.608918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.608912    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wf5p\" (UniqueName: \"kubernetes.io/projected/07a6ee17-7736-44a8-8a55-9ad954129b8b-kube-api-access-2wf5p\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:00.962800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.962759    5112 generic.go:356] "Generic (PLEG): container finished" podID="07a6ee17-7736-44a8-8a55-9ad954129b8b" containerID="4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342" exitCode=0
Nov 03 16:47:00.962994 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.962812    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"
Nov 03 16:47:00.963050 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.962807    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9" event={"ID":"07a6ee17-7736-44a8-8a55-9ad954129b8b","Type":"ContainerDied","Data":"4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342"}
Nov 03 16:47:00.963100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.963051    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9" event={"ID":"07a6ee17-7736-44a8-8a55-9ad954129b8b","Type":"ContainerDied","Data":"cc4eb98ccae86cef76c31c479e460f0e07e01b9daebc18f811c729b701dd44ef"}
Nov 03 16:47:00.963100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.963083    5112 scope.go:122] "RemoveContainer" containerID="4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342"
Nov 03 16:47:00.977058 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.977032    5112 scope.go:122] "RemoveContainer" containerID="4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342"
Nov 03 16:47:00.979467 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:47:00.979436    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342\": container with ID starting with 4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342 not found: ID does not exist" containerID="4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342"
Nov 03 16:47:00.979625 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:00.979471    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342"} err="failed to get container status \"4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342\": rpc error: code = NotFound desc = could not find container \"4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342\": container with ID starting with 4b65b282937d27be699869808748ec1ad860d36e0c3e0467af72d4c625877342 not found: ID does not exist"
Nov 03 16:47:01.135719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:01.135686    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"]
Nov 03 16:47:01.149612 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:01.149569    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9635db0d-56b778ffbc-6f8z9"]
Nov 03 16:47:02.553061 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:02.553017    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a6ee17-7736-44a8-8a55-9ad954129b8b" path="/var/lib/kubelet/pods/07a6ee17-7736-44a8-8a55-9ad954129b8b/volumes"
Nov 03 16:47:07.887462 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:47:07.887419    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:47:11.797885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:11.797847    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"]
Nov 03 16:47:11.798411 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:11.798115    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v" podUID="6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" containerName="maas-api" containerID="cri-o://c976585d3cb66107f0b949e702725d6d694726749de3b1781a913d362dadde7d" gracePeriod=30
Nov 03 16:47:12.102756 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.102641    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"]
Nov 03 16:47:12.120897 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.120022    5112 generic.go:356] "Generic (PLEG): container finished" podID="6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" containerID="c976585d3cb66107f0b949e702725d6d694726749de3b1781a913d362dadde7d" exitCode=0
Nov 03 16:47:12.120897 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.120122    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v" event={"ID":"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7","Type":"ContainerDied","Data":"c976585d3cb66107f0b949e702725d6d694726749de3b1781a913d362dadde7d"}
Nov 03 16:47:12.159827 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.159799    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:47:12.309122 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.309083    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-metrics-certs\") pod \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") "
Nov 03 16:47:12.309122 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.309130    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls\") pod \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") "
Nov 03 16:47:12.309427 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.309361    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-kube-api-access-j4c9r\" (UniqueName: \"kubernetes.io/projected/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-kube-api-access-j4c9r\") pod \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") "
Nov 03 16:47:12.309486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.309466    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-openshift-service-ca\") pod \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\" (UID: \"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7\") "
Nov 03 16:47:12.309853 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.309824    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-openshift-service-ca" pod "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" (UID: "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:47:12.310429 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.310410    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:12.311735 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.311710    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-kube-api-access-j4c9r" pod "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" (UID: "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7"). InnerVolumeSpecName "kube-api-access-j4c9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:47:12.311825 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.311736    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls" pod "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" (UID: "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:47:12.311894 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.311876    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-metrics-certs" pod "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" (UID: "6835a522-d79e-4a1c-8dbf-8a6d04dc28e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:47:12.412012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.411975    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4c9r\" (UniqueName: \"kubernetes.io/projected/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-kube-api-access-j4c9r\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:12.412012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.412014    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:12.412293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:12.412031    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:13.129062 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:13.129029    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"
Nov 03 16:47:13.129062 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:13.129042    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v" event={"ID":"6835a522-d79e-4a1c-8dbf-8a6d04dc28e7","Type":"ContainerDied","Data":"52c50840047a59d2bdc6b4d6f72254692104e5a92d842cc4680b9ff8fa93da0b"}
Nov 03 16:47:13.129694 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:13.129108    5112 scope.go:122] "RemoveContainer" containerID="c976585d3cb66107f0b949e702725d6d694726749de3b1781a913d362dadde7d"
Nov 03 16:47:13.294257 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:13.294218    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"]
Nov 03 16:47:13.339303 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:13.339250    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6822cb-74f98947-p9v5v"]
Nov 03 16:47:14.551741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:14.551704    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" path="/var/lib/kubelet/pods/6835a522-d79e-4a1c-8dbf-8a6d04dc28e7/volumes"
Nov 03 16:47:17.108138 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.107726    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq" podUID="c3d3d850-986d-4dbf-9854-a8c647328cb8" containerName="payload-pre-processing" containerID="cri-o://9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4" gracePeriod=25
Nov 03 16:47:17.427739 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.424812    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:47:17.542184 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.542152    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d3d850-986d-4dbf-9854-a8c647328cb8-kube-api-access-zhncv\" (UniqueName: \"kubernetes.io/projected/c3d3d850-986d-4dbf-9854-a8c647328cb8-kube-api-access-zhncv\") pod \"c3d3d850-986d-4dbf-9854-a8c647328cb8\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") "
Nov 03 16:47:17.542353 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.542291    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d3d850-986d-4dbf-9854-a8c647328cb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d3d850-986d-4dbf-9854-a8c647328cb8-plugins-config-volume\") pod \"c3d3d850-986d-4dbf-9854-a8c647328cb8\" (UID: \"c3d3d850-986d-4dbf-9854-a8c647328cb8\") "
Nov 03 16:47:17.543321 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.543276    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d3d850-986d-4dbf-9854-a8c647328cb8-plugins-config-volume" pod "c3d3d850-986d-4dbf-9854-a8c647328cb8" (UID: "c3d3d850-986d-4dbf-9854-a8c647328cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:47:17.545377 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.545349    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d3d850-986d-4dbf-9854-a8c647328cb8-kube-api-access-zhncv" pod "c3d3d850-986d-4dbf-9854-a8c647328cb8" (UID: "c3d3d850-986d-4dbf-9854-a8c647328cb8"). InnerVolumeSpecName "kube-api-access-zhncv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:47:17.644574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.644545    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhncv\" (UniqueName: \"kubernetes.io/projected/c3d3d850-986d-4dbf-9854-a8c647328cb8-kube-api-access-zhncv\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:17.644703 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:17.644580    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d3d850-986d-4dbf-9854-a8c647328cb8-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:18.172280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.172241    5112 generic.go:356] "Generic (PLEG): container finished" podID="c3d3d850-986d-4dbf-9854-a8c647328cb8" containerID="9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4" exitCode=0
Nov 03 16:47:18.172280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.172276    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"
Nov 03 16:47:18.172814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.172288    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq" event={"ID":"c3d3d850-986d-4dbf-9854-a8c647328cb8","Type":"ContainerDied","Data":"9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4"}
Nov 03 16:47:18.172814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.172346    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq" event={"ID":"c3d3d850-986d-4dbf-9854-a8c647328cb8","Type":"ContainerDied","Data":"252785832b923a33251e545be91ee52cbefc615ec9efe21388d29964cbba4fef"}
Nov 03 16:47:18.172814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.172372    5112 scope.go:122] "RemoveContainer" containerID="9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4"
Nov 03 16:47:18.185703 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.185683    5112 scope.go:122] "RemoveContainer" containerID="9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4"
Nov 03 16:47:18.185985 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:47:18.185966    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4\": container with ID starting with 9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4 not found: ID does not exist" containerID="9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4"
Nov 03 16:47:18.186068 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.185990    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4"} err="failed to get container status \"9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4\": rpc error: code = NotFound desc = could not find container \"9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4\": container with ID starting with 9729891201a3495237003d50e05ca978a97dec65a060f6cf12ec03573251f3f4 not found: ID does not exist"
Nov 03 16:47:18.340843 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.340813    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"]
Nov 03 16:47:18.416222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.406912    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6822cb-64cf54dbd7-fjfpq"]
Nov 03 16:47:18.677165 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:18.677118    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d3d850-986d-4dbf-9854-a8c647328cb8" path="/var/lib/kubelet/pods/c3d3d850-986d-4dbf-9854-a8c647328cb8/volumes"
Nov 03 16:47:37.885132 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:47:37.885095    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:47:37.980087 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:37.980051    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"]
Nov 03 16:47:45.125483 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:45.125443    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"]
Nov 03 16:47:45.257433 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:45.257382    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"]
Nov 03 16:47:50.126412 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:50.126364    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" podUID="59f795ab-eafb-46bd-969d-551771cdbd7a" containerName="payload-processing" containerID="cri-o://bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098" gracePeriod=25
Nov 03 16:47:50.258850 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:50.258811    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" podUID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" containerName="payload-pre-processing" containerID="cri-o://0d460b439bd147f49b2714f7608e4d6b0789d70a847d197b7580b298fe49a8b0" gracePeriod=25
Nov 03 16:47:52.248446 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:52.248409    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" podUID="59f795ab-eafb-46bd-969d-551771cdbd7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:47:52.248833 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:52.248468    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" podUID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:47:53.002395 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.002351    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main" containerID="cri-o://ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28" gracePeriod=45
Nov 03 16:47:53.321576 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.321552    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:47:53.411860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.411826    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-home\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-home\") pod \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") "
Nov 03 16:47:53.412043 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.411909    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tls-certs\") pod \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") "
Nov 03 16:47:53.412043 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.411974    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfebe23a-01a1-4b27-ad8d-2497166eef3a-kube-api-access-rk7jh\" (UniqueName: \"kubernetes.io/projected/cfebe23a-01a1-4b27-ad8d-2497166eef3a-kube-api-access-rk7jh\") pod \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") "
Nov 03 16:47:53.412043 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.412012    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-dshm\") pod \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") "
Nov 03 16:47:53.412276 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.412092    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tmp-dir\") pod \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") "
Nov 03 16:47:53.412276 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.412158    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-model-cache\") pod \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\" (UID: \"cfebe23a-01a1-4b27-ad8d-2497166eef3a\") "
Nov 03 16:47:53.413156 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.413126    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-model-cache" pod "cfebe23a-01a1-4b27-ad8d-2497166eef3a" (UID: "cfebe23a-01a1-4b27-ad8d-2497166eef3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:47:53.413399 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.413377    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-home" pod "cfebe23a-01a1-4b27-ad8d-2497166eef3a" (UID: "cfebe23a-01a1-4b27-ad8d-2497166eef3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:47:53.414005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.413976    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tmp-dir" pod "cfebe23a-01a1-4b27-ad8d-2497166eef3a" (UID: "cfebe23a-01a1-4b27-ad8d-2497166eef3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:47:53.417070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.417032    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tls-certs" pod "cfebe23a-01a1-4b27-ad8d-2497166eef3a" (UID: "cfebe23a-01a1-4b27-ad8d-2497166eef3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:47:53.417181 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.417057    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfebe23a-01a1-4b27-ad8d-2497166eef3a-kube-api-access-rk7jh" pod "cfebe23a-01a1-4b27-ad8d-2497166eef3a" (UID: "cfebe23a-01a1-4b27-ad8d-2497166eef3a"). InnerVolumeSpecName "kube-api-access-rk7jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:47:53.417353 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.417321    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-dshm" pod "cfebe23a-01a1-4b27-ad8d-2497166eef3a" (UID: "cfebe23a-01a1-4b27-ad8d-2497166eef3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:47:53.513917 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.513885    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:53.513917 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.513918    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:53.514131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.513933    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk7jh\" (UniqueName: \"kubernetes.io/projected/cfebe23a-01a1-4b27-ad8d-2497166eef3a-kube-api-access-rk7jh\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:53.514131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.513948    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:53.514131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.513959    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:53.514131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.513970    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfebe23a-01a1-4b27-ad8d-2497166eef3a-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:47:53.519326 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.519298    5112 generic.go:356] "Generic (PLEG): container finished" podID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerID="ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28" exitCode=0
Nov 03 16:47:53.519450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.519351    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"
Nov 03 16:47:53.519450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.519386    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" event={"ID":"cfebe23a-01a1-4b27-ad8d-2497166eef3a","Type":"ContainerDied","Data":"ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28"}
Nov 03 16:47:53.519450 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.519436    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk" event={"ID":"cfebe23a-01a1-4b27-ad8d-2497166eef3a","Type":"ContainerDied","Data":"6faaaa6d8f804e5fbc0a7660701fa8d63ca97e73bf236f9bc167498a0612d656"}
Nov 03 16:47:53.519594 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.519460    5112 scope.go:122] "RemoveContainer" containerID="ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28"
Nov 03 16:47:53.532829 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.532812    5112 scope.go:122] "RemoveContainer" containerID="ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28"
Nov 03 16:47:53.533092 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:47:53.533075    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28\": container with ID starting with ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28 not found: ID does not exist" containerID="ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28"
Nov 03 16:47:53.533127 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.533102    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28"} err="failed to get container status \"ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28\": rpc error: code = NotFound desc = could not find container \"ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28\": container with ID starting with ea9565e756d85c741c204670eb511290c47f49020e7a372f82a3595592858c28 not found: ID does not exist"
Nov 03 16:47:53.681504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.681473    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"]
Nov 03 16:47:53.688280 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:53.688253    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2bb620/distinct-w4-2bb620-kserve-65576bf8dc-vv8fk"]
Nov 03 16:47:54.551046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:47:54.551015    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" path="/var/lib/kubelet/pods/cfebe23a-01a1-4b27-ad8d-2497166eef3a/volumes"
Nov 03 16:48:02.247943 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:02.247902    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" podUID="59f795ab-eafb-46bd-969d-551771cdbd7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:48:02.248376 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:02.247914    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" podUID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:48:07.886023 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:48:07.885964    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:48:10.724477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.724396    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:48:10.749424 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.749385    5112 generic.go:356] "Generic (PLEG): container finished" podID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" containerID="0d460b439bd147f49b2714f7608e4d6b0789d70a847d197b7580b298fe49a8b0" exitCode=0
Nov 03 16:48:10.749937 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.749905    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" event={"ID":"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08","Type":"ContainerDied","Data":"0d460b439bd147f49b2714f7608e4d6b0789d70a847d197b7580b298fe49a8b0"}
Nov 03 16:48:10.751964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.751941    5112 generic.go:356] "Generic (PLEG): container finished" podID="59f795ab-eafb-46bd-969d-551771cdbd7a" containerID="bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098" exitCode=0
Nov 03 16:48:10.752213 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.752180    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" event={"ID":"59f795ab-eafb-46bd-969d-551771cdbd7a","Type":"ContainerDied","Data":"bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098"}
Nov 03 16:48:10.752349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.752334    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27" event={"ID":"59f795ab-eafb-46bd-969d-551771cdbd7a","Type":"ContainerDied","Data":"5354fbb9072f935a448231c3a435e6b785dc0cd87d6c4230afee481e32273da8"}
Nov 03 16:48:10.752453 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.752442    5112 scope.go:122] "RemoveContainer" containerID="bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098"
Nov 03 16:48:10.752691 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.752677    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"
Nov 03 16:48:10.754804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.754782    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f795ab-eafb-46bd-969d-551771cdbd7a-kube-api-access-6pv69\" (UniqueName: \"kubernetes.io/projected/59f795ab-eafb-46bd-969d-551771cdbd7a-kube-api-access-6pv69\") pod \"59f795ab-eafb-46bd-969d-551771cdbd7a\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") "
Nov 03 16:48:10.755053 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.755034    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f795ab-eafb-46bd-969d-551771cdbd7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f795ab-eafb-46bd-969d-551771cdbd7a-plugins-config-volume\") pod \"59f795ab-eafb-46bd-969d-551771cdbd7a\" (UID: \"59f795ab-eafb-46bd-969d-551771cdbd7a\") "
Nov 03 16:48:10.756266 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.756242    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f795ab-eafb-46bd-969d-551771cdbd7a-plugins-config-volume" pod "59f795ab-eafb-46bd-969d-551771cdbd7a" (UID: "59f795ab-eafb-46bd-969d-551771cdbd7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:48:10.772349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.772317    5112 scope.go:122] "RemoveContainer" containerID="bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098"
Nov 03 16:48:10.772746 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:48:10.772720    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098\": container with ID starting with bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098 not found: ID does not exist" containerID="bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098"
Nov 03 16:48:10.772847 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.772756    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098"} err="failed to get container status \"bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098\": rpc error: code = NotFound desc = could not find container \"bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098\": container with ID starting with bc83762cec2f5216c7912c895dfe955fef61d192762023bb62d2e2ea4b0f9098 not found: ID does not exist"
Nov 03 16:48:10.791445 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.791388    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f795ab-eafb-46bd-969d-551771cdbd7a-kube-api-access-6pv69" pod "59f795ab-eafb-46bd-969d-551771cdbd7a" (UID: "59f795ab-eafb-46bd-969d-551771cdbd7a"). InnerVolumeSpecName "kube-api-access-6pv69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:48:10.871842 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.871803    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f795ab-eafb-46bd-969d-551771cdbd7a-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:48:10.872081 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.872062    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pv69\" (UniqueName: \"kubernetes.io/projected/59f795ab-eafb-46bd-969d-551771cdbd7a-kube-api-access-6pv69\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:48:10.914915 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.914884    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:48:10.990806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:10.990703    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-plugins-config-volume\") pod \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") "
Nov 03 16:48:11.020220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.002687    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-plugins-config-volume" pod "63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" (UID: "63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:48:11.020220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.016756    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-kube-api-access-zvcp5\" (UniqueName: \"kubernetes.io/projected/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-kube-api-access-zvcp5\") pod \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\" (UID: \"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08\") "
Nov 03 16:48:11.020220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.017686    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:48:11.026304 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.024601    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-kube-api-access-zvcp5" pod "63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" (UID: "63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08"). InnerVolumeSpecName "kube-api-access-zvcp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:48:11.120030 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.119986    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvcp5\" (UniqueName: \"kubernetes.io/projected/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08-kube-api-access-zvcp5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:48:11.195513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.195463    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"]
Nov 03 16:48:11.201132 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.201103    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2bb620-5987cf8975-47c27"]
Nov 03 16:48:11.765531 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.765488    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6" event={"ID":"63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08","Type":"ContainerDied","Data":"c2e09a13fcdbfb9f004653508edca9b160ee939cbda108cf14dcbaa4e5618884"}
Nov 03 16:48:11.766014 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.765570    5112 scope.go:122] "RemoveContainer" containerID="0d460b439bd147f49b2714f7608e4d6b0789d70a847d197b7580b298fe49a8b0"
Nov 03 16:48:11.766014 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.765509    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"
Nov 03 16:48:11.944293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.944250    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"]
Nov 03 16:48:11.988225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:11.987725    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2bb620-6fc5485bf5-4lxj6"]
Nov 03 16:48:12.565225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:12.564415    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f795ab-eafb-46bd-969d-551771cdbd7a" path="/var/lib/kubelet/pods/59f795ab-eafb-46bd-969d-551771cdbd7a/volumes"
Nov 03 16:48:12.570224 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:12.568007    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" path="/var/lib/kubelet/pods/63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08/volumes"
Nov 03 16:48:37.885379 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:48:37.885341    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:48:39.011447 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:39.011404    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"]
Nov 03 16:48:44.012583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.012544    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4" podUID="d37bc61b-61dc-4b3b-94a3-bbc54babfca8" containerName="payload-pre-processing" containerID="cri-o://d651623c9092b006694f46a3a4c92d7c651992ea51969e777ba8cfbfbe9363f6" gracePeriod=25
Nov 03 16:48:44.167376 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.167344    5112 generic.go:356] "Generic (PLEG): container finished" podID="d37bc61b-61dc-4b3b-94a3-bbc54babfca8" containerID="d651623c9092b006694f46a3a4c92d7c651992ea51969e777ba8cfbfbe9363f6" exitCode=0
Nov 03 16:48:44.167548 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.167398    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4" event={"ID":"d37bc61b-61dc-4b3b-94a3-bbc54babfca8","Type":"ContainerDied","Data":"d651623c9092b006694f46a3a4c92d7c651992ea51969e777ba8cfbfbe9363f6"}
Nov 03 16:48:44.341771 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.341747    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:48:44.352110 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.352078    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-plugins-config-volume\") pod \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") "
Nov 03 16:48:44.352298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.352192    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-kube-api-access-c6fpt\" (UniqueName: \"kubernetes.io/projected/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-kube-api-access-c6fpt\") pod \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\" (UID: \"d37bc61b-61dc-4b3b-94a3-bbc54babfca8\") "
Nov 03 16:48:44.352472 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.352449    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-plugins-config-volume" pod "d37bc61b-61dc-4b3b-94a3-bbc54babfca8" (UID: "d37bc61b-61dc-4b3b-94a3-bbc54babfca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:48:44.352762 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.352744    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:48:44.354337 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.354307    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-kube-api-access-c6fpt" pod "d37bc61b-61dc-4b3b-94a3-bbc54babfca8" (UID: "d37bc61b-61dc-4b3b-94a3-bbc54babfca8"). InnerVolumeSpecName "kube-api-access-c6fpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:48:44.454539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:44.454513    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fpt\" (UniqueName: \"kubernetes.io/projected/d37bc61b-61dc-4b3b-94a3-bbc54babfca8-kube-api-access-c6fpt\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:48:45.175679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:45.175643    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4" event={"ID":"d37bc61b-61dc-4b3b-94a3-bbc54babfca8","Type":"ContainerDied","Data":"7ff587123462b0fca1b16685cd79a419e2dacbd4b3b6a68fa1b6ab8336532957"}
Nov 03 16:48:45.175679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:45.175676    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"
Nov 03 16:48:45.176284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:45.175880    5112 scope.go:122] "RemoveContainer" containerID="d651623c9092b006694f46a3a4c92d7c651992ea51969e777ba8cfbfbe9363f6"
Nov 03 16:48:45.343112 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:45.343072    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"]
Nov 03 16:48:45.348740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:45.348713    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0ea8a5-9d6b5bf7f-jktb4"]
Nov 03 16:48:46.602923 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:48:46.602884    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37bc61b-61dc-4b3b-94a3-bbc54babfca8" path="/var/lib/kubelet/pods/d37bc61b-61dc-4b3b-94a3-bbc54babfca8/volumes"
Nov 03 16:49:07.884906 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:49:07.884868    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:49:23.388784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.388749    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"]
Nov 03 16:49:23.389927 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.388944    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" containerID="cri-o://b4f5ff099ca39e772ec7c533596b23bb2b3882b3b126f1e2dc03e1f9147d5075" gracePeriod=30
Nov 03 16:49:23.401411 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.401380    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:49:23.401657 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.401632    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:49:23.560232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.560161    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"]
Nov 03 16:49:23.560402 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.560380    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" containerID="cri-o://1773357816be7565e32e148df2759cc2de3a3ccb86a24608082c6a39cb003968" gracePeriod=30
Nov 03 16:49:23.875335 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.875302    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"]
Nov 03 16:49:23.875538 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:23.875514    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" containerID="cri-o://57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b" gracePeriod=30
Nov 03 16:49:24.460332 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:24.460297    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"]
Nov 03 16:49:24.460828 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:24.460500    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" containerID="cri-o://8d717528f832c2631431b799c46b9008b12036123897867e269116d526e41c0c" gracePeriod=30
Nov 03 16:49:25.510160 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:25.510118    5112 patch_prober.go:29] interesting pod/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:49:25.510559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:25.510188    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:49:27.417578 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:27.417236    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:49:27.417578 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:27.417298    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:49:27.671912 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:27.671829    5112 patch_prober.go:29] interesting pod/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:49:27.671912 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:27.671894    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:49:28.616094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.616068    5112 generic.go:356] "Generic (PLEG): container finished" podID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerID="1773357816be7565e32e148df2759cc2de3a3ccb86a24608082c6a39cb003968" exitCode=0
Nov 03 16:49:28.616567 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.616546    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" event={"ID":"5c0f5855-bc6a-4677-93ce-dd659bf590d7","Type":"ContainerDied","Data":"1773357816be7565e32e148df2759cc2de3a3ccb86a24608082c6a39cb003968"}
Nov 03 16:49:28.618146 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.618121    5112 generic.go:356] "Generic (PLEG): container finished" podID="681ca214-8021-47b1-b94f-73e47fea2010" containerID="b4f5ff099ca39e772ec7c533596b23bb2b3882b3b126f1e2dc03e1f9147d5075" exitCode=0
Nov 03 16:49:28.618284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.618219    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" event={"ID":"681ca214-8021-47b1-b94f-73e47fea2010","Type":"ContainerDied","Data":"b4f5ff099ca39e772ec7c533596b23bb2b3882b3b126f1e2dc03e1f9147d5075"}
Nov 03 16:49:28.719860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.719572    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:49:28.796758 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.796729    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:49:28.835764 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.835737    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681ca214-8021-47b1-b94f-73e47fea2010-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681ca214-8021-47b1-b94f-73e47fea2010-istiod-ca-cert\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.835903 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.835784    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-credential-socket\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.835903 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.835865    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/681ca214-8021-47b1-b94f-73e47fea2010-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681ca214-8021-47b1-b94f-73e47fea2010-istio-podinfo\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.835903 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.835888    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-token\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-token\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.835949    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-envoy\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.835999    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-socket\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836033    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-data\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.836100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836073    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-podinfo\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836102    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-data\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836117    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-credential-socket" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.836237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836147    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-certs\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836193    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istiod-ca-cert\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836296    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-socket\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.836468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836341    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-certs\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.836468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836380    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-kube-api-access-4lglh\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-kube-api-access-4lglh\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836359    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-data" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.836468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836414    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-istio-token\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-istio-token\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.836468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836424    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-socket" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.836468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836463    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-kube-api-access-svdz6\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-kube-api-access-svdz6\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.836841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836478    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681ca214-8021-47b1-b94f-73e47fea2010-istiod-ca-cert" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:49:28.836841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836593    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-envoy\") pod \"681ca214-8021-47b1-b94f-73e47fea2010\" (UID: \"681ca214-8021-47b1-b94f-73e47fea2010\") "
Nov 03 16:49:28.836841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836633    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-credential-socket\") pod \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\" (UID: \"5c0f5855-bc6a-4677-93ce-dd659bf590d7\") "
Nov 03 16:49:28.836841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836721    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-data" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.836841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836746    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-certs" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.836841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836803    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istiod-ca-cert" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:49:28.837235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.836960    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-socket" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.837235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837000    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-credential-socket" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.837359 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837316    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-certs" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.837428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837397    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837424    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681ca214-8021-47b1-b94f-73e47fea2010-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837440    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837455    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837467    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837480    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837733 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837494    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837733 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837507    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.837733 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.837522    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.839602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839574    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-envoy" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.839716 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839634    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-envoy" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:28.839869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839847    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/681ca214-8021-47b1-b94f-73e47fea2010-istio-podinfo" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:49:28.839958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839860    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-podinfo" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:49:28.839958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839888    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-istio-token" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:28.839958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839927    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-kube-api-access-svdz6" pod "681ca214-8021-47b1-b94f-73e47fea2010" (UID: "681ca214-8021-47b1-b94f-73e47fea2010"). InnerVolumeSpecName "kube-api-access-svdz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:28.839958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839923    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-kube-api-access-4lglh" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "kube-api-access-4lglh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:28.839958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.839945    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-token" pod "5c0f5855-bc6a-4677-93ce-dd659bf590d7" (UID: "5c0f5855-bc6a-4677-93ce-dd659bf590d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:28.939582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939502    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.939582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939541    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681ca214-8021-47b1-b94f-73e47fea2010-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.939582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939557    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.939582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939571    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.939582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939584    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0f5855-bc6a-4677-93ce-dd659bf590d7-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.940019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939598    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681ca214-8021-47b1-b94f-73e47fea2010-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.940019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939611    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lglh\" (UniqueName: \"kubernetes.io/projected/5c0f5855-bc6a-4677-93ce-dd659bf590d7-kube-api-access-4lglh\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.940019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939626    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:28.940019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:28.939639    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svdz6\" (UniqueName: \"kubernetes.io/projected/681ca214-8021-47b1-b94f-73e47fea2010-kube-api-access-svdz6\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.170624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.170595    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:49:29.244885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.244801    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-credential-socket\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.244885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.244837    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-data\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.244885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.244883    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-kube-api-access-8ls66\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-kube-api-access-8ls66\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.245154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.244911    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-token\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-token\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.245154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.244972    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-socket\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.245154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245088    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-certs\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.245154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245132    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istiod-ca-cert\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.245154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245146    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-data" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.245463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245243    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-podinfo\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.245463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245249    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-credential-socket" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.245463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245262    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-envoy\") pod \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\" (UID: \"dca07c6b-d01a-403d-81d6-2dbebcbd8c13\") "
Nov 03 16:49:29.245463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245370    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-certs" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.245463 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245390    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-socket" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.246477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.245571    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istiod-ca-cert" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:49:29.246477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.246114    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.246477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.246141    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.246477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.246158    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.246477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.246172    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.246477 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.246186    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.247387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.247366    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-token" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:29.247387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.247375    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-envoy" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.247628 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.247612    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-podinfo" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:49:29.247669 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.247642    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-kube-api-access-8ls66" pod "dca07c6b-d01a-403d-81d6-2dbebcbd8c13" (UID: "dca07c6b-d01a-403d-81d6-2dbebcbd8c13"). InnerVolumeSpecName "kube-api-access-8ls66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:29.348059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.348026    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.348059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.348056    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.348059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.348066    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ls66\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-kube-api-access-8ls66\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.348319 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.348076    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca07c6b-d01a-403d-81d6-2dbebcbd8c13-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.629149 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.629115    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"
Nov 03 16:49:29.629627 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.629143    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5" event={"ID":"681ca214-8021-47b1-b94f-73e47fea2010","Type":"ContainerDied","Data":"776ddc51c681dce3271a85920b714dccc319068a9cfdfc39a8155e53b7507a4a"}
Nov 03 16:49:29.629627 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.629230    5112 scope.go:122] "RemoveContainer" containerID="b4f5ff099ca39e772ec7c533596b23bb2b3882b3b126f1e2dc03e1f9147d5075"
Nov 03 16:49:29.631074 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.631044    5112 generic.go:356] "Generic (PLEG): container finished" podID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerID="57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b" exitCode=0
Nov 03 16:49:29.631236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.631091    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"
Nov 03 16:49:29.631236 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.631156    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" event={"ID":"dca07c6b-d01a-403d-81d6-2dbebcbd8c13","Type":"ContainerDied","Data":"57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b"}
Nov 03 16:49:29.631357 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.631302    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68" event={"ID":"dca07c6b-d01a-403d-81d6-2dbebcbd8c13","Type":"ContainerDied","Data":"7580d11fda6d12f685547298d99ba95c36f5146412d82765134d3019540c3278"}
Nov 03 16:49:29.633924 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.633185    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94" event={"ID":"5c0f5855-bc6a-4677-93ce-dd659bf590d7","Type":"ContainerDied","Data":"415ed70eecfb46ae5d65850f41d2f24e6d1b023226221478abb6e39cd78fd6c5"}
Nov 03 16:49:29.633924 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.633312    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"
Nov 03 16:49:29.639177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.639154    5112 generic.go:356] "Generic (PLEG): container finished" podID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerID="8d717528f832c2631431b799c46b9008b12036123897867e269116d526e41c0c" exitCode=0
Nov 03 16:49:29.639299 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.639272    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" event={"ID":"a16acfc1-e819-4a7c-b026-c7ffde68da12","Type":"ContainerDied","Data":"8d717528f832c2631431b799c46b9008b12036123897867e269116d526e41c0c"}
Nov 03 16:49:29.646497 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.644948    5112 scope.go:122] "RemoveContainer" containerID="57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b"
Nov 03 16:49:29.679438 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.666646    5112 scope.go:122] "RemoveContainer" containerID="57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b"
Nov 03 16:49:29.679438 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:49:29.667044    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b\": container with ID starting with 57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b not found: ID does not exist" containerID="57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b"
Nov 03 16:49:29.679438 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.667078    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b"} err="failed to get container status \"57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b\": rpc error: code = NotFound desc = could not find container \"57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b\": container with ID starting with 57e58667763319a2daa361c4d08845d29124e942741e6fdcdce6a71225686b9b not found: ID does not exist"
Nov 03 16:49:29.679438 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.667104    5112 scope.go:122] "RemoveContainer" containerID="1773357816be7565e32e148df2759cc2de3a3ccb86a24608082c6a39cb003968"
Nov 03 16:49:29.752119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.752060    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:49:29.796871 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.796838    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"]
Nov 03 16:49:29.802440 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.802412    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-mvw94"]
Nov 03 16:49:29.862624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862585    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-envoy\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.862806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862650    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-socket\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.862806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862681    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-token\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-token\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.862806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862739    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-podinfo\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.862806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862772    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-credential-socket\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.863020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862824    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-certs\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.863020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862979    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-data\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.863119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863034    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16acfc1-e819-4a7c-b026-c7ffde68da12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16acfc1-e819-4a7c-b026-c7ffde68da12-istiod-ca-cert\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.863119 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863109    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-kube-api-access-j7r28\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-kube-api-access-j7r28\") pod \"a16acfc1-e819-4a7c-b026-c7ffde68da12\" (UID: \"a16acfc1-e819-4a7c-b026-c7ffde68da12\") "
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.862972    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-socket" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863225    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-data" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863241    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-credential-socket" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863485    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-certs" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863596    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16acfc1-e819-4a7c-b026-c7ffde68da12-istiod-ca-cert" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863836    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863863    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16acfc1-e819-4a7c-b026-c7ffde68da12-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863879    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863894    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.864804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.863910    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.865967 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.865941    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-podinfo" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:49:29.866092 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.865983    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-envoy" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:29.866298 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.866271    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-kube-api-access-j7r28" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "kube-api-access-j7r28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:29.867499 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.867464    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-token" pod "a16acfc1-e819-4a7c-b026-c7ffde68da12" (UID: "a16acfc1-e819-4a7c-b026-c7ffde68da12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:29.950827 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.950786    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"]
Nov 03 16:49:29.955805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.955777    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-2rt68"]
Nov 03 16:49:29.965140 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.965110    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7r28\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-kube-api-access-j7r28\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.965140 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.965137    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.965338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.965149    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:29.965338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:29.965159    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16acfc1-e819-4a7c-b026-c7ffde68da12-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:30.066098 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.066009    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"]
Nov 03 16:49:30.072627 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.072594    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xwrn5"]
Nov 03 16:49:30.551269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.551232    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" path="/var/lib/kubelet/pods/5c0f5855-bc6a-4677-93ce-dd659bf590d7/volumes"
Nov 03 16:49:30.551872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.551854    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681ca214-8021-47b1-b94f-73e47fea2010" path="/var/lib/kubelet/pods/681ca214-8021-47b1-b94f-73e47fea2010/volumes"
Nov 03 16:49:30.552383 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.552369    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" path="/var/lib/kubelet/pods/dca07c6b-d01a-403d-81d6-2dbebcbd8c13/volumes"
Nov 03 16:49:30.648690 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.648663    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"
Nov 03 16:49:30.649193 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.648702    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl" event={"ID":"a16acfc1-e819-4a7c-b026-c7ffde68da12","Type":"ContainerDied","Data":"c480b9ea29f681f380919addbebb6abe5961412f2f0bc73b64b4f6ef8184dee3"}
Nov 03 16:49:30.649193 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.648774    5112 scope.go:122] "RemoveContainer" containerID="8d717528f832c2631431b799c46b9008b12036123897867e269116d526e41c0c"
Nov 03 16:49:30.814407 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.814313    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"]
Nov 03 16:49:30.862439 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:30.862406    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g4xkl"]
Nov 03 16:49:32.551347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:32.551316    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" path="/var/lib/kubelet/pods/a16acfc1-e819-4a7c-b026-c7ffde68da12/volumes"
Nov 03 16:49:35.122793 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:35.122761    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"]
Nov 03 16:49:37.762696 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:37.762645    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"]
Nov 03 16:49:37.763249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:37.762938    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy" containerID="cri-o://cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7" gracePeriod=30
Nov 03 16:49:37.945334 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:49:37.945293    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:49:42.808470 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:42.808401    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"]
Nov 03 16:49:42.809345 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:42.809300    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25" podUID="a00cbb13-70ee-40a0-a1f8-f4982b375909" containerName="maas-api" containerID="cri-o://63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e" gracePeriod=30
Nov 03 16:49:43.246956 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.246923    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:49:43.271951 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.271923    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:49:43.304013 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.303986    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-istio-token\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-istio-token\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304176 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304155    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-socket\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304243 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304190    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe61a3c-41f2-4688-9886-3cc710efb308-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe61a3c-41f2-4688-9886-3cc710efb308-istiod-ca-cert\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304243 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304236    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls\") pod \"a00cbb13-70ee-40a0-a1f8-f4982b375909\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") "
Nov 03 16:49:43.304364 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304277    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00cbb13-70ee-40a0-a1f8-f4982b375909-kube-api-access-dbn7t\" (UniqueName: \"kubernetes.io/projected/a00cbb13-70ee-40a0-a1f8-f4982b375909-kube-api-access-dbn7t\") pod \"a00cbb13-70ee-40a0-a1f8-f4982b375909\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") "
Nov 03 16:49:43.304452 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304414    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-credential-socket\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304454    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-kube-api-access-f7h6m\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-kube-api-access-f7h6m\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304517 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304494    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-socket" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:43.304630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304533    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abe61a3c-41f2-4688-9886-3cc710efb308-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe61a3c-41f2-4688-9886-3cc710efb308-istio-podinfo\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304569    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-metrics-certs\") pod \"a00cbb13-70ee-40a0-a1f8-f4982b375909\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") "
Nov 03 16:49:43.304749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304628    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-envoy\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304673    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-data\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.304749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.304684    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe61a3c-41f2-4688-9886-3cc710efb308-istiod-ca-cert" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:49:43.305055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.305029    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-data" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:43.305184 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.305086    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-credential-socket" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:43.305350 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.305330    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.305413 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.305356    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.305413 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.305373    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.305413 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.305388    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe61a3c-41f2-4688-9886-3cc710efb308-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.307486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.307427    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-metrics-certs" pod "a00cbb13-70ee-40a0-a1f8-f4982b375909" (UID: "a00cbb13-70ee-40a0-a1f8-f4982b375909"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:49:43.307680 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.307654    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abe61a3c-41f2-4688-9886-3cc710efb308-istio-podinfo" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:49:43.307740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.307659    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-kube-api-access-f7h6m" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "kube-api-access-f7h6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:43.307740 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.307701    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00cbb13-70ee-40a0-a1f8-f4982b375909-kube-api-access-dbn7t" pod "a00cbb13-70ee-40a0-a1f8-f4982b375909" (UID: "a00cbb13-70ee-40a0-a1f8-f4982b375909"). InnerVolumeSpecName "kube-api-access-dbn7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:43.307849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.307801    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-istio-token" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:43.307849 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.307835    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-envoy" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:43.308603 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.308581    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls" pod "a00cbb13-70ee-40a0-a1f8-f4982b375909" (UID: "a00cbb13-70ee-40a0-a1f8-f4982b375909"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:49:43.406404 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.406370    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00cbb13-70ee-40a0-a1f8-f4982b375909-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00cbb13-70ee-40a0-a1f8-f4982b375909-openshift-service-ca\") pod \"a00cbb13-70ee-40a0-a1f8-f4982b375909\" (UID: \"a00cbb13-70ee-40a0-a1f8-f4982b375909\") "
Nov 03 16:49:43.406632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.406423    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-certs\") pod \"abe61a3c-41f2-4688-9886-3cc710efb308\" (UID: \"abe61a3c-41f2-4688-9886-3cc710efb308\") "
Nov 03 16:49:43.406806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.406777    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-certs" pod "abe61a3c-41f2-4688-9886-3cc710efb308" (UID: "abe61a3c-41f2-4688-9886-3cc710efb308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:43.406806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.406791    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00cbb13-70ee-40a0-a1f8-f4982b375909-openshift-service-ca" pod "a00cbb13-70ee-40a0-a1f8-f4982b375909" (UID: "a00cbb13-70ee-40a0-a1f8-f4982b375909"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:49:43.406998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.406978    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00cbb13-70ee-40a0-a1f8-f4982b375909-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407008    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407025    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407039    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407053    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbn7t\" (UniqueName: \"kubernetes.io/projected/a00cbb13-70ee-40a0-a1f8-f4982b375909-kube-api-access-dbn7t\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407068    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7h6m\" (UniqueName: \"kubernetes.io/projected/abe61a3c-41f2-4688-9886-3cc710efb308-kube-api-access-f7h6m\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407083    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe61a3c-41f2-4688-9886-3cc710efb308-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407102 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407097    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00cbb13-70ee-40a0-a1f8-f4982b375909-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.407309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.407110    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe61a3c-41f2-4688-9886-3cc710efb308-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:43.819748 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.819657    5112 generic.go:356] "Generic (PLEG): container finished" podID="a00cbb13-70ee-40a0-a1f8-f4982b375909" containerID="63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e" exitCode=0
Nov 03 16:49:43.819748 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.819707    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"
Nov 03 16:49:43.820312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.819810    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25" event={"ID":"a00cbb13-70ee-40a0-a1f8-f4982b375909","Type":"ContainerDied","Data":"63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e"}
Nov 03 16:49:43.820312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.819861    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25" event={"ID":"a00cbb13-70ee-40a0-a1f8-f4982b375909","Type":"ContainerDied","Data":"e6f547f5399ccfade6b11b7e752e1786264861eef755a7a4e6f2d6690d53e1e1"}
Nov 03 16:49:43.820312 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.819892    5112 scope.go:122] "RemoveContainer" containerID="63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e"
Nov 03 16:49:43.822267 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.822227    5112 generic.go:356] "Generic (PLEG): container finished" podID="abe61a3c-41f2-4688-9886-3cc710efb308" containerID="cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7" exitCode=0
Nov 03 16:49:43.822536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.822324    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"
Nov 03 16:49:43.822626 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.822357    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" event={"ID":"abe61a3c-41f2-4688-9886-3cc710efb308","Type":"ContainerDied","Data":"cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7"}
Nov 03 16:49:43.822736 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.822686    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n" event={"ID":"abe61a3c-41f2-4688-9886-3cc710efb308","Type":"ContainerDied","Data":"4f436302311f924acb3d5191b1657000212a5088804e13c8e3049ccb634d86d2"}
Nov 03 16:49:43.836550 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.836526    5112 scope.go:122] "RemoveContainer" containerID="63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e"
Nov 03 16:49:43.836924 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:49:43.836893    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e\": container with ID starting with 63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e not found: ID does not exist" containerID="63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e"
Nov 03 16:49:43.837038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.836931    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e"} err="failed to get container status \"63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e\": rpc error: code = NotFound desc = could not find container \"63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e\": container with ID starting with 63475074137cd3f7e1d953ef6e7d33805d68156b8c385571b0f4e4f14f0f195e not found: ID does not exist"
Nov 03 16:49:43.837038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.836948    5112 scope.go:122] "RemoveContainer" containerID="cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7"
Nov 03 16:49:43.927620 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.926643    5112 scope.go:122] "RemoveContainer" containerID="cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7"
Nov 03 16:49:43.927620 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:49:43.927044    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7\": container with ID starting with cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7 not found: ID does not exist" containerID="cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7"
Nov 03 16:49:43.927620 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:43.927078    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7"} err="failed to get container status \"cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7\": rpc error: code = NotFound desc = could not find container \"cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7\": container with ID starting with cf53fbac795a43aa9b3e4446a32a2a07009f0c595d05b96143cd03561c780bf7 not found: ID does not exist"
Nov 03 16:49:44.071231 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:44.070276    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"]
Nov 03 16:49:44.106557 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:44.106512    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2bb620-openshift-default-5f5cd474fc-vx64n"]
Nov 03 16:49:44.185279 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:44.180992    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"]
Nov 03 16:49:44.197042 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:44.196996    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0ea8a5-86578669c9-r6l25"]
Nov 03 16:49:44.551115 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:44.551081    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00cbb13-70ee-40a0-a1f8-f4982b375909" path="/var/lib/kubelet/pods/a00cbb13-70ee-40a0-a1f8-f4982b375909/volumes"
Nov 03 16:49:44.551652 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:44.551633    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" path="/var/lib/kubelet/pods/abe61a3c-41f2-4688-9886-3cc710efb308/volumes"
Nov 03 16:49:49.059881 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.059693    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"]
Nov 03 16:49:49.060609 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.060575    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk" podUID="8677da00-1571-4dbf-ad73-6432017e538e" containerName="maas-api" containerID="cri-o://6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef" gracePeriod=30
Nov 03 16:49:49.404662 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.404636    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:49:49.502686 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.502642    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls\") pod \"8677da00-1571-4dbf-ad73-6432017e538e\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") "
Nov 03 16:49:49.502884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.502733    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8677da00-1571-4dbf-ad73-6432017e538e-kube-api-access-sqvh2\" (UniqueName: \"kubernetes.io/projected/8677da00-1571-4dbf-ad73-6432017e538e-kube-api-access-sqvh2\") pod \"8677da00-1571-4dbf-ad73-6432017e538e\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") "
Nov 03 16:49:49.502952 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.502900    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-metrics-certs\") pod \"8677da00-1571-4dbf-ad73-6432017e538e\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") "
Nov 03 16:49:49.503007 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.502947    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8677da00-1571-4dbf-ad73-6432017e538e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8677da00-1571-4dbf-ad73-6432017e538e-openshift-service-ca\") pod \"8677da00-1571-4dbf-ad73-6432017e538e\" (UID: \"8677da00-1571-4dbf-ad73-6432017e538e\") "
Nov 03 16:49:49.503824 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.503791    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8677da00-1571-4dbf-ad73-6432017e538e-openshift-service-ca" pod "8677da00-1571-4dbf-ad73-6432017e538e" (UID: "8677da00-1571-4dbf-ad73-6432017e538e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:49:49.505963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.505934    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-metrics-certs" pod "8677da00-1571-4dbf-ad73-6432017e538e" (UID: "8677da00-1571-4dbf-ad73-6432017e538e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:49:49.506388 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.506359    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls" pod "8677da00-1571-4dbf-ad73-6432017e538e" (UID: "8677da00-1571-4dbf-ad73-6432017e538e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:49:49.507240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.507168    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8677da00-1571-4dbf-ad73-6432017e538e-kube-api-access-sqvh2" pod "8677da00-1571-4dbf-ad73-6432017e538e" (UID: "8677da00-1571-4dbf-ad73-6432017e538e"). InnerVolumeSpecName "kube-api-access-sqvh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:49.605125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.605084    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:49.605125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.605119    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvh2\" (UniqueName: \"kubernetes.io/projected/8677da00-1571-4dbf-ad73-6432017e538e-kube-api-access-sqvh2\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:49.605354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.605135    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8677da00-1571-4dbf-ad73-6432017e538e-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:49.605354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.605148    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8677da00-1571-4dbf-ad73-6432017e538e-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:49.870002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.869964    5112 generic.go:356] "Generic (PLEG): container finished" podID="8677da00-1571-4dbf-ad73-6432017e538e" containerID="6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef" exitCode=0
Nov 03 16:49:49.870177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.870030    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"
Nov 03 16:49:49.870177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.870051    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk" event={"ID":"8677da00-1571-4dbf-ad73-6432017e538e","Type":"ContainerDied","Data":"6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef"}
Nov 03 16:49:49.870177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.870112    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk" event={"ID":"8677da00-1571-4dbf-ad73-6432017e538e","Type":"ContainerDied","Data":"55752e861a464c7b43fb3f4f416094d76e69685ae050702b00048041c30bfa66"}
Nov 03 16:49:49.870177 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.870141    5112 scope.go:122] "RemoveContainer" containerID="6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef"
Nov 03 16:49:49.883101 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.883079    5112 scope.go:122] "RemoveContainer" containerID="6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef"
Nov 03 16:49:49.883381 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:49:49.883362    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef\": container with ID starting with 6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef not found: ID does not exist" containerID="6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef"
Nov 03 16:49:49.883470 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:49.883388    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef"} err="failed to get container status \"6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef\": rpc error: code = NotFound desc = could not find container \"6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef\": container with ID starting with 6c1087d999509289886948e3aa704a5f05418f7e06bd896fa350128b43d645ef not found: ID does not exist"
Nov 03 16:49:50.044727 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.040826    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"]
Nov 03 16:49:50.094223 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.081652    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0fda73-79cc7b5b49-f98jk"]
Nov 03 16:49:50.144499 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.144463    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main" containerID="cri-o://6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3" gracePeriod=45
Nov 03 16:49:50.452251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.452225    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:49:50.518123 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518089    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348e1e8e-9cc9-4c31-9097-b766fe181877-kube-api-access-jfdr9\" (UniqueName: \"kubernetes.io/projected/348e1e8e-9cc9-4c31-9097-b766fe181877-kube-api-access-jfdr9\") pod \"348e1e8e-9cc9-4c31-9097-b766fe181877\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") "
Nov 03 16:49:50.518360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518176    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-model-cache\") pod \"348e1e8e-9cc9-4c31-9097-b766fe181877\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") "
Nov 03 16:49:50.518360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518242    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-dshm\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-dshm\") pod \"348e1e8e-9cc9-4c31-9097-b766fe181877\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") "
Nov 03 16:49:50.518360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518271    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348e1e8e-9cc9-4c31-9097-b766fe181877-tls-certs\" (UniqueName: \"kubernetes.io/secret/348e1e8e-9cc9-4c31-9097-b766fe181877-tls-certs\") pod \"348e1e8e-9cc9-4c31-9097-b766fe181877\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") "
Nov 03 16:49:50.518360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518300    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-tmp-dir\") pod \"348e1e8e-9cc9-4c31-9097-b766fe181877\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") "
Nov 03 16:49:50.518360 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518327    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-home\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-home\") pod \"348e1e8e-9cc9-4c31-9097-b766fe181877\" (UID: \"348e1e8e-9cc9-4c31-9097-b766fe181877\") "
Nov 03 16:49:50.518630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518469    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-model-cache" pod "348e1e8e-9cc9-4c31-9097-b766fe181877" (UID: "348e1e8e-9cc9-4c31-9097-b766fe181877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:50.518630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518544    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-tmp-dir" pod "348e1e8e-9cc9-4c31-9097-b766fe181877" (UID: "348e1e8e-9cc9-4c31-9097-b766fe181877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:50.518630 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518591    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-home" pod "348e1e8e-9cc9-4c31-9097-b766fe181877" (UID: "348e1e8e-9cc9-4c31-9097-b766fe181877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:50.518864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518846    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:50.518925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518867    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:50.518925 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.518880    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:50.520511 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.520490    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348e1e8e-9cc9-4c31-9097-b766fe181877-kube-api-access-jfdr9" pod "348e1e8e-9cc9-4c31-9097-b766fe181877" (UID: "348e1e8e-9cc9-4c31-9097-b766fe181877"). InnerVolumeSpecName "kube-api-access-jfdr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:49:50.520771 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.520749    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-dshm" pod "348e1e8e-9cc9-4c31-9097-b766fe181877" (UID: "348e1e8e-9cc9-4c31-9097-b766fe181877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:49:50.520771 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.520766    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348e1e8e-9cc9-4c31-9097-b766fe181877-tls-certs" pod "348e1e8e-9cc9-4c31-9097-b766fe181877" (UID: "348e1e8e-9cc9-4c31-9097-b766fe181877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:49:50.551247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.551217    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8677da00-1571-4dbf-ad73-6432017e538e" path="/var/lib/kubelet/pods/8677da00-1571-4dbf-ad73-6432017e538e/volumes"
Nov 03 16:49:50.620695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.620655    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348e1e8e-9cc9-4c31-9097-b766fe181877-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:50.620695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.620690    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfdr9\" (UniqueName: \"kubernetes.io/projected/348e1e8e-9cc9-4c31-9097-b766fe181877-kube-api-access-jfdr9\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:50.620695 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.620700    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348e1e8e-9cc9-4c31-9097-b766fe181877-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:49:50.881632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.881545    5112 generic.go:356] "Generic (PLEG): container finished" podID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerID="6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3" exitCode=0
Nov 03 16:49:50.881632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.881583    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"
Nov 03 16:49:50.881632 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.881608    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" event={"ID":"348e1e8e-9cc9-4c31-9097-b766fe181877","Type":"ContainerDied","Data":"6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3"}
Nov 03 16:49:50.881888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.881651    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk" event={"ID":"348e1e8e-9cc9-4c31-9097-b766fe181877","Type":"ContainerDied","Data":"8ce2185a5cf802961860e95742db12329948c86a2073afd46d3f0a3e26c1007f"}
Nov 03 16:49:50.881888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.881674    5112 scope.go:122] "RemoveContainer" containerID="6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3"
Nov 03 16:49:50.892852 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.892823    5112 scope.go:122] "RemoveContainer" containerID="6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3"
Nov 03 16:49:50.893142 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:49:50.893121    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3\": container with ID starting with 6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3 not found: ID does not exist" containerID="6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3"
Nov 03 16:49:50.893240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:50.893152    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3"} err="failed to get container status \"6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3\": rpc error: code = NotFound desc = could not find container \"6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3\": container with ID starting with 6d00286638696ce7e63097849ffd761f918b72c5782264b1956a045ca70fbea3 not found: ID does not exist"
Nov 03 16:49:51.055498 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:51.055453    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"]
Nov 03 16:49:51.065785 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:51.065755    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-547a38/premium-w6-547a38-kserve-789d4bc846-7tzlk"]
Nov 03 16:49:52.569073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:52.569034    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" path="/var/lib/kubelet/pods/348e1e8e-9cc9-4c31-9097-b766fe181877/volumes"
Nov 03 16:49:59.937617 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:59.937578    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"]
Nov 03 16:49:59.986771 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:49:59.986730    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"]
Nov 03 16:50:07.889461 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:50:07.889411    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:50:09.623464 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:09.623430    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"]
Nov 03 16:50:09.740708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:09.740670    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"]
Nov 03 16:50:13.919677 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:13.919644    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"]
Nov 03 16:50:14.044784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:14.044738    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"]
Nov 03 16:50:14.624667 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:14.624628    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8" podUID="f710392e-71fa-4bcf-af87-96ec3ea904c3" containerName="payload-processing" containerID="cri-o://e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e" gracePeriod=25
Nov 03 16:50:14.741405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:14.741345    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" podUID="c0866a99-1153-465a-90d6-06a647a8fda6" containerName="payload-pre-processing" containerID="cri-o://d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1" gracePeriod=25
Nov 03 16:50:14.949967 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:14.949943    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:50:14.961277 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:14.961232    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" containerID="cri-o://72a38ab8f40f1176cbce942e5c1be6059e64ae9fa6cca2b1b85fe24ff5279869" gracePeriod=45
Nov 03 16:50:15.008293 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.008257    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main" containerID="cri-o://b1fd8192b3fded90e832dda40ed01f09f6dd1b474e3860146656158f2073729d" gracePeriod=45
Nov 03 16:50:15.096823 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.096792    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f710392e-71fa-4bcf-af87-96ec3ea904c3-kube-api-access-7sxsk\" (UniqueName: \"kubernetes.io/projected/f710392e-71fa-4bcf-af87-96ec3ea904c3-kube-api-access-7sxsk\") pod \"f710392e-71fa-4bcf-af87-96ec3ea904c3\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") "
Nov 03 16:50:15.097351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.096892    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f710392e-71fa-4bcf-af87-96ec3ea904c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f710392e-71fa-4bcf-af87-96ec3ea904c3-plugins-config-volume\") pod \"f710392e-71fa-4bcf-af87-96ec3ea904c3\" (UID: \"f710392e-71fa-4bcf-af87-96ec3ea904c3\") "
Nov 03 16:50:15.097513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.097460    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f710392e-71fa-4bcf-af87-96ec3ea904c3-plugins-config-volume" pod "f710392e-71fa-4bcf-af87-96ec3ea904c3" (UID: "f710392e-71fa-4bcf-af87-96ec3ea904c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:50:15.100070 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.099941    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f710392e-71fa-4bcf-af87-96ec3ea904c3-kube-api-access-7sxsk" pod "f710392e-71fa-4bcf-af87-96ec3ea904c3" (UID: "f710392e-71fa-4bcf-af87-96ec3ea904c3"). InnerVolumeSpecName "kube-api-access-7sxsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:15.121254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.121168    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"]
Nov 03 16:50:15.126721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.126683    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a433e7-v6xkw"]
Nov 03 16:50:15.198352 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.198322    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sxsk\" (UniqueName: \"kubernetes.io/projected/f710392e-71fa-4bcf-af87-96ec3ea904c3-kube-api-access-7sxsk\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.198352 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.198352    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f710392e-71fa-4bcf-af87-96ec3ea904c3-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.225039 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.224998    5112 generic.go:356] "Generic (PLEG): container finished" podID="f710392e-71fa-4bcf-af87-96ec3ea904c3" containerID="e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e" exitCode=0
Nov 03 16:50:15.225173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.225044    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"
Nov 03 16:50:15.225173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.225075    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8" event={"ID":"f710392e-71fa-4bcf-af87-96ec3ea904c3","Type":"ContainerDied","Data":"e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e"}
Nov 03 16:50:15.225173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.225131    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8" event={"ID":"f710392e-71fa-4bcf-af87-96ec3ea904c3","Type":"ContainerDied","Data":"2564a17ff0274824974b4839e7f70d22343bd6723e72003e59e65694ad230cc6"}
Nov 03 16:50:15.225173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.225156    5112 scope.go:122] "RemoveContainer" containerID="e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e"
Nov 03 16:50:15.229145 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.229120    5112 generic.go:356] "Generic (PLEG): container finished" podID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerID="72a38ab8f40f1176cbce942e5c1be6059e64ae9fa6cca2b1b85fe24ff5279869" exitCode=0
Nov 03 16:50:15.229301 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.229266    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" event={"ID":"1474dd53-f607-4a36-880a-670a21b9a1d8","Type":"ContainerDied","Data":"72a38ab8f40f1176cbce942e5c1be6059e64ae9fa6cca2b1b85fe24ff5279869"}
Nov 03 16:50:15.231846 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.231744    5112 generic.go:356] "Generic (PLEG): container finished" podID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerID="b1fd8192b3fded90e832dda40ed01f09f6dd1b474e3860146656158f2073729d" exitCode=0
Nov 03 16:50:15.231846 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.231805    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" event={"ID":"95d8b397-8b41-42c3-8bdf-8ef23ee8663d","Type":"ContainerDied","Data":"b1fd8192b3fded90e832dda40ed01f09f6dd1b474e3860146656158f2073729d"}
Nov 03 16:50:15.243536 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.243520    5112 scope.go:122] "RemoveContainer" containerID="e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e"
Nov 03 16:50:15.243901 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:50:15.243855    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e\": container with ID starting with e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e not found: ID does not exist" containerID="e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e"
Nov 03 16:50:15.243991 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.243913    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e"} err="failed to get container status \"e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e\": rpc error: code = NotFound desc = could not find container \"e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e\": container with ID starting with e4c96bc6fadb6381a115b8b16e78157791ad7d1df616449bbd7957582ef3851e not found: ID does not exist"
Nov 03 16:50:15.310074 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.310043    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:50:15.325674 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.325650    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:50:15.396208 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.396142    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"]
Nov 03 16:50:15.402991 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.402932    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-192564-74997b84bc-pfsw8"]
Nov 03 16:50:15.404599 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.404562    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-tmp-dir\") pod \"1474dd53-f607-4a36-880a-670a21b9a1d8\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") "
Nov 03 16:50:15.404806 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.404784    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-dshm\") pod \"1474dd53-f607-4a36-880a-670a21b9a1d8\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") "
Nov 03 16:50:15.404896 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.404838    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-tmp-dir" pod "1474dd53-f607-4a36-880a-670a21b9a1d8" (UID: "1474dd53-f607-4a36-880a-670a21b9a1d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.404896 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.404871    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-home\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-home\") pod \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") "
Nov 03 16:50:15.405009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.404922    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-home\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-home\") pod \"1474dd53-f607-4a36-880a-670a21b9a1d8\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") "
Nov 03 16:50:15.405009 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.404989    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-dshm\") pod \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") "
Nov 03 16:50:15.405320 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.405300    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-home" pod "1474dd53-f607-4a36-880a-670a21b9a1d8" (UID: "1474dd53-f607-4a36-880a-670a21b9a1d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.405824 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.405803    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-home" pod "95d8b397-8b41-42c3-8bdf-8ef23ee8663d" (UID: "95d8b397-8b41-42c3-8bdf-8ef23ee8663d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.405923 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.405892    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tmp-dir\") pod \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") "
Nov 03 16:50:15.405986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.405923    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tls-certs\" (UniqueName: \"kubernetes.io/secret/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tls-certs\") pod \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") "
Nov 03 16:50:15.405986 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.405961    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-kube-api-access-6vrg8\" (UniqueName: \"kubernetes.io/projected/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-kube-api-access-6vrg8\") pod \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") "
Nov 03 16:50:15.406095 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.406036    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1474dd53-f607-4a36-880a-670a21b9a1d8-kube-api-access-6rk7z\" (UniqueName: \"kubernetes.io/projected/1474dd53-f607-4a36-880a-670a21b9a1d8-kube-api-access-6rk7z\") pod \"1474dd53-f607-4a36-880a-670a21b9a1d8\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") "
Nov 03 16:50:15.406150 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.406101    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-model-cache\") pod \"1474dd53-f607-4a36-880a-670a21b9a1d8\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") "
Nov 03 16:50:15.406260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.406150    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-model-cache\") pod \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\" (UID: \"95d8b397-8b41-42c3-8bdf-8ef23ee8663d\") "
Nov 03 16:50:15.406260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.406251    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1474dd53-f607-4a36-880a-670a21b9a1d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/1474dd53-f607-4a36-880a-670a21b9a1d8-tls-certs\") pod \"1474dd53-f607-4a36-880a-670a21b9a1d8\" (UID: \"1474dd53-f607-4a36-880a-670a21b9a1d8\") "
Nov 03 16:50:15.406863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.406842    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.407011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.406865    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.407011 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.406879    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.407672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.407648    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-dshm" pod "1474dd53-f607-4a36-880a-670a21b9a1d8" (UID: "1474dd53-f607-4a36-880a-670a21b9a1d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.408331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.407946    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-model-cache" pod "1474dd53-f607-4a36-880a-670a21b9a1d8" (UID: "1474dd53-f607-4a36-880a-670a21b9a1d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.408331 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.408295    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-model-cache" pod "95d8b397-8b41-42c3-8bdf-8ef23ee8663d" (UID: "95d8b397-8b41-42c3-8bdf-8ef23ee8663d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.408938 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.408917    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tmp-dir" pod "95d8b397-8b41-42c3-8bdf-8ef23ee8663d" (UID: "95d8b397-8b41-42c3-8bdf-8ef23ee8663d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.409473 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.409452    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1474dd53-f607-4a36-880a-670a21b9a1d8-tls-certs" pod "1474dd53-f607-4a36-880a-670a21b9a1d8" (UID: "1474dd53-f607-4a36-880a-670a21b9a1d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:50:15.410577 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.410542    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1474dd53-f607-4a36-880a-670a21b9a1d8-kube-api-access-6rk7z" pod "1474dd53-f607-4a36-880a-670a21b9a1d8" (UID: "1474dd53-f607-4a36-880a-670a21b9a1d8"). InnerVolumeSpecName "kube-api-access-6rk7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:15.411286 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.411262    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tls-certs" pod "95d8b397-8b41-42c3-8bdf-8ef23ee8663d" (UID: "95d8b397-8b41-42c3-8bdf-8ef23ee8663d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:50:15.412038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.412003    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-dshm" pod "95d8b397-8b41-42c3-8bdf-8ef23ee8663d" (UID: "95d8b397-8b41-42c3-8bdf-8ef23ee8663d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:50:15.412801 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.412780    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-kube-api-access-6vrg8" pod "95d8b397-8b41-42c3-8bdf-8ef23ee8663d" (UID: "95d8b397-8b41-42c3-8bdf-8ef23ee8663d"). InnerVolumeSpecName "kube-api-access-6vrg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:15.508554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508479    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rk7z\" (UniqueName: \"kubernetes.io/projected/1474dd53-f607-4a36-880a-670a21b9a1d8-kube-api-access-6rk7z\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508507    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508519    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508532    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1474dd53-f607-4a36-880a-670a21b9a1d8-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508542    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1474dd53-f607-4a36-880a-670a21b9a1d8-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508551    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508559    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508878 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508568    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:15.508878 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:15.508575    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrg8\" (UniqueName: \"kubernetes.io/projected/95d8b397-8b41-42c3-8bdf-8ef23ee8663d-kube-api-access-6vrg8\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:16.218627 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.218581    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 16:50:16.240002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.239974    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"
Nov 03 16:50:16.240217 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.240005    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz" event={"ID":"1474dd53-f607-4a36-880a-670a21b9a1d8","Type":"ContainerDied","Data":"c0f1f623cc6db1ea202b292dfd30c9115fe8bb9bfabec45fefe13d49240df085"}
Nov 03 16:50:16.240217 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.240057    5112 scope.go:122] "RemoveContainer" containerID="72a38ab8f40f1176cbce942e5c1be6059e64ae9fa6cca2b1b85fe24ff5279869"
Nov 03 16:50:16.241665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.241646    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"
Nov 03 16:50:16.241788 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.241694    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls" event={"ID":"95d8b397-8b41-42c3-8bdf-8ef23ee8663d","Type":"ContainerDied","Data":"83c0d2770a893a0c9a35102c0da51393428d7f34b2814887f6c613a643e335eb"}
Nov 03 16:50:16.251067 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.251043    5112 scope.go:122] "RemoveContainer" containerID="b1fd8192b3fded90e832dda40ed01f09f6dd1b474e3860146656158f2073729d"
Nov 03 16:50:16.408347 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.408315    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"]
Nov 03 16:50:16.411623 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.411594    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-w3-192564-kserve-6bb594bdb7-66mls"]
Nov 03 16:50:16.520296 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.520212    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"]
Nov 03 16:50:16.530446 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.530409    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-192564/distinct-2-w3-192564-kserve-68b47cb9cb-rsdsz"]
Nov 03 16:50:16.552130 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.552098    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" path="/var/lib/kubelet/pods/1474dd53-f607-4a36-880a-670a21b9a1d8/volumes"
Nov 03 16:50:16.552648 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.552627    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" path="/var/lib/kubelet/pods/95d8b397-8b41-42c3-8bdf-8ef23ee8663d/volumes"
Nov 03 16:50:16.553120 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.553100    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c094c8be-971f-42ce-b2b2-5b05f36a94a6" path="/var/lib/kubelet/pods/c094c8be-971f-42ce-b2b2-5b05f36a94a6/volumes"
Nov 03 16:50:16.553911 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:16.553876    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f710392e-71fa-4bcf-af87-96ec3ea904c3" path="/var/lib/kubelet/pods/f710392e-71fa-4bcf-af87-96ec3ea904c3/volumes"
Nov 03 16:50:17.214086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:17.214047    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" podUID="c0866a99-1153-465a-90d6-06a647a8fda6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:50:18.461373 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.461339    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"]
Nov 03 16:50:18.462079 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.462051    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t" podUID="53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" containerName="maas-api" containerID="cri-o://bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b" gracePeriod=30
Nov 03 16:50:18.799308 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.799278    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:50:18.864085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.864045    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-kube-api-access-bjcfh\" (UniqueName: \"kubernetes.io/projected/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-kube-api-access-bjcfh\") pod \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") "
Nov 03 16:50:18.864349 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.864109    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-openshift-service-ca\") pod \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") "
Nov 03 16:50:18.864486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.864389    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls\") pod \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") "
Nov 03 16:50:18.864486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.864431    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-metrics-certs\") pod \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\" (UID: \"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6\") "
Nov 03 16:50:18.865885 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.865850    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-openshift-service-ca" pod "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" (UID: "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:50:18.869250 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.869186    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-metrics-certs" pod "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" (UID: "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:50:18.871006 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.870976    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls" pod "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" (UID: "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:50:18.871879 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.871855    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-kube-api-access-bjcfh" pod "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" (UID: "53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6"). InnerVolumeSpecName "kube-api-access-bjcfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:18.920618 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.920556    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" podUID="88b706ed-062e-40a3-8044-bafc0c131f07" containerName="payload-processing" containerID="cri-o://c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28" gracePeriod=25
Nov 03 16:50:18.965974 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.965930    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:18.965974 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.965972    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:18.966286 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.965995    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcfh\" (UniqueName: \"kubernetes.io/projected/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-kube-api-access-bjcfh\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:18.966286 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:18.966011    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:19.046008 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.045703    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" podUID="b08b03b9-4124-4294-bab1-29e4c4c85917" containerName="payload-pre-processing" containerID="cri-o://66dccaa33192035c1168e82b9d5f0761f96af7fe1cc76218db395a2831c0b84c" gracePeriod=25
Nov 03 16:50:19.269616 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.269579    5112 generic.go:356] "Generic (PLEG): container finished" podID="53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" containerID="bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b" exitCode=0
Nov 03 16:50:19.269863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.269645    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"
Nov 03 16:50:19.269863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.269659    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t" event={"ID":"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6","Type":"ContainerDied","Data":"bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b"}
Nov 03 16:50:19.269863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.269725    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t" event={"ID":"53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6","Type":"ContainerDied","Data":"08a7ae204ce5df6b81c1fcbaefd4e96b4f021a69e9a55bb81f9abebb30946c9d"}
Nov 03 16:50:19.269863 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.269759    5112 scope.go:122] "RemoveContainer" containerID="bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b"
Nov 03 16:50:19.282539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.282514    5112 scope.go:122] "RemoveContainer" containerID="bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b"
Nov 03 16:50:19.282820 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:50:19.282800    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b\": container with ID starting with bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b not found: ID does not exist" containerID="bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b"
Nov 03 16:50:19.282891 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.282827    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b"} err="failed to get container status \"bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b\": rpc error: code = NotFound desc = could not find container \"bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b\": container with ID starting with bfed22fddfb9ea00084651e889c619b7155bfc9d7a9451b6a64da76895446b8b not found: ID does not exist"
Nov 03 16:50:19.446219 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.443738    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"]
Nov 03 16:50:19.449678 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:19.449645    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66f145-76cf795c7b-9gz5t"]
Nov 03 16:50:20.551699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:20.551666    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" path="/var/lib/kubelet/pods/53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6/volumes"
Nov 03 16:50:22.167408 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:22.167376    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" podUID="88b706ed-062e-40a3-8044-bafc0c131f07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:50:22.167809 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:22.167379    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" podUID="b08b03b9-4124-4294-bab1-29e4c4c85917" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:50:27.214217 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:27.214167    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" podUID="c0866a99-1153-465a-90d6-06a647a8fda6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:50:32.166975 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:32.166923    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" podUID="88b706ed-062e-40a3-8044-bafc0c131f07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:50:32.167556 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:32.167022    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" podUID="b08b03b9-4124-4294-bab1-29e4c4c85917" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 16:50:35.048290 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.048260    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:50:35.151346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.151308    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0866a99-1153-465a-90d6-06a647a8fda6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866a99-1153-465a-90d6-06a647a8fda6-plugins-config-volume\") pod \"c0866a99-1153-465a-90d6-06a647a8fda6\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") "
Nov 03 16:50:35.151548 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.151531    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0866a99-1153-465a-90d6-06a647a8fda6-kube-api-access-vbxnh\" (UniqueName: \"kubernetes.io/projected/c0866a99-1153-465a-90d6-06a647a8fda6-kube-api-access-vbxnh\") pod \"c0866a99-1153-465a-90d6-06a647a8fda6\" (UID: \"c0866a99-1153-465a-90d6-06a647a8fda6\") "
Nov 03 16:50:35.151657 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.151631    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0866a99-1153-465a-90d6-06a647a8fda6-plugins-config-volume" pod "c0866a99-1153-465a-90d6-06a647a8fda6" (UID: "c0866a99-1153-465a-90d6-06a647a8fda6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:50:35.151958 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.151939    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866a99-1153-465a-90d6-06a647a8fda6-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:35.153793 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.153767    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0866a99-1153-465a-90d6-06a647a8fda6-kube-api-access-vbxnh" pod "c0866a99-1153-465a-90d6-06a647a8fda6" (UID: "c0866a99-1153-465a-90d6-06a647a8fda6"). InnerVolumeSpecName "kube-api-access-vbxnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:35.292333 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.289065    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbxnh\" (UniqueName: \"kubernetes.io/projected/c0866a99-1153-465a-90d6-06a647a8fda6-kube-api-access-vbxnh\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:35.405639 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.405597    5112 generic.go:356] "Generic (PLEG): container finished" podID="c0866a99-1153-465a-90d6-06a647a8fda6" containerID="d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1" exitCode=0
Nov 03 16:50:35.405844 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.405662    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"
Nov 03 16:50:35.405844 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.405665    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" event={"ID":"c0866a99-1153-465a-90d6-06a647a8fda6","Type":"ContainerDied","Data":"d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1"}
Nov 03 16:50:35.405985 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.405894    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d" event={"ID":"c0866a99-1153-465a-90d6-06a647a8fda6","Type":"ContainerDied","Data":"079223ceca241aad7f9460dbaf1b975094693805c47118752e1239460f1d4480"}
Nov 03 16:50:35.405985 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.405937    5112 scope.go:122] "RemoveContainer" containerID="d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1"
Nov 03 16:50:35.420503 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.420481    5112 scope.go:122] "RemoveContainer" containerID="d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1"
Nov 03 16:50:35.420881 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:50:35.420858    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1\": container with ID starting with d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1 not found: ID does not exist" containerID="d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1"
Nov 03 16:50:35.420946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.420893    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1"} err="failed to get container status \"d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1\": rpc error: code = NotFound desc = could not find container \"d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1\": container with ID starting with d80721214ace6363ce2a70b0974a05bdd5ef80806bbfdd75e676ed7cc269ebf1 not found: ID does not exist"
Nov 03 16:50:35.602998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.602916    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"]
Nov 03 16:50:35.607692 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:35.607663    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-192564-7974b76679-h4n7d"]
Nov 03 16:50:36.553901 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:36.553862    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0866a99-1153-465a-90d6-06a647a8fda6" path="/var/lib/kubelet/pods/c0866a99-1153-465a-90d6-06a647a8fda6/volumes"
Nov 03 16:50:37.888798 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:50:37.888762    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:50:39.133316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.133170    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:50:39.133316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.133174    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:50:39.157681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.157571    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:50:39.173240 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.167072    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:50:39.399473 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.399417    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:50:39.443666 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.443624    5112 generic.go:356] "Generic (PLEG): container finished" podID="b08b03b9-4124-4294-bab1-29e4c4c85917" containerID="66dccaa33192035c1168e82b9d5f0761f96af7fe1cc76218db395a2831c0b84c" exitCode=0
Nov 03 16:50:39.443864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.443795    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" event={"ID":"b08b03b9-4124-4294-bab1-29e4c4c85917","Type":"ContainerDied","Data":"66dccaa33192035c1168e82b9d5f0761f96af7fe1cc76218db395a2831c0b84c"}
Nov 03 16:50:39.445559 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.445527    5112 generic.go:356] "Generic (PLEG): container finished" podID="88b706ed-062e-40a3-8044-bafc0c131f07" containerID="c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28" exitCode=0
Nov 03 16:50:39.445741 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.445709    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" event={"ID":"88b706ed-062e-40a3-8044-bafc0c131f07","Type":"ContainerDied","Data":"c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28"}
Nov 03 16:50:39.445822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.445749    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl" event={"ID":"88b706ed-062e-40a3-8044-bafc0c131f07","Type":"ContainerDied","Data":"a31ba38b17988e95c826e0440ba59b7fe61aa04cb239aa0a55107e82592d276d"}
Nov 03 16:50:39.445822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.445778    5112 scope.go:122] "RemoveContainer" containerID="c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28"
Nov 03 16:50:39.445964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.445935    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"
Nov 03 16:50:39.537612 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.534416    5112 scope.go:122] "RemoveContainer" containerID="c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28"
Nov 03 16:50:39.546644 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:50:39.546608    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28\": container with ID starting with c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28 not found: ID does not exist" containerID="c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28"
Nov 03 16:50:39.551835 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.548281    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28"} err="failed to get container status \"c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28\": rpc error: code = NotFound desc = could not find container \"c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28\": container with ID starting with c7c0db4b16cc5d5820ab69b0865cf3aeb7eef17d341c0a733009c423d9809a28 not found: ID does not exist"
Nov 03 16:50:39.565379 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.565344    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:50:39.571846 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.571012    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b706ed-062e-40a3-8044-bafc0c131f07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b706ed-062e-40a3-8044-bafc0c131f07-plugins-config-volume\") pod \"88b706ed-062e-40a3-8044-bafc0c131f07\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") "
Nov 03 16:50:39.571846 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.571135    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b706ed-062e-40a3-8044-bafc0c131f07-kube-api-access-2nmrt\" (UniqueName: \"kubernetes.io/projected/88b706ed-062e-40a3-8044-bafc0c131f07-kube-api-access-2nmrt\") pod \"88b706ed-062e-40a3-8044-bafc0c131f07\" (UID: \"88b706ed-062e-40a3-8044-bafc0c131f07\") "
Nov 03 16:50:39.574461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.574395    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b706ed-062e-40a3-8044-bafc0c131f07-plugins-config-volume" pod "88b706ed-062e-40a3-8044-bafc0c131f07" (UID: "88b706ed-062e-40a3-8044-bafc0c131f07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:50:39.574461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.574419    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b706ed-062e-40a3-8044-bafc0c131f07-kube-api-access-2nmrt" pod "88b706ed-062e-40a3-8044-bafc0c131f07" (UID: "88b706ed-062e-40a3-8044-bafc0c131f07"). InnerVolumeSpecName "kube-api-access-2nmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:39.672237 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.672178    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08b03b9-4124-4294-bab1-29e4c4c85917-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08b03b9-4124-4294-bab1-29e4c4c85917-plugins-config-volume\") pod \"b08b03b9-4124-4294-bab1-29e4c4c85917\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") "
Nov 03 16:50:39.672457 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.672380    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08b03b9-4124-4294-bab1-29e4c4c85917-kube-api-access-rq9t8\" (UniqueName: \"kubernetes.io/projected/b08b03b9-4124-4294-bab1-29e4c4c85917-kube-api-access-rq9t8\") pod \"b08b03b9-4124-4294-bab1-29e4c4c85917\" (UID: \"b08b03b9-4124-4294-bab1-29e4c4c85917\") "
Nov 03 16:50:39.672652 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.672622    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08b03b9-4124-4294-bab1-29e4c4c85917-plugins-config-volume" pod "b08b03b9-4124-4294-bab1-29e4c4c85917" (UID: "b08b03b9-4124-4294-bab1-29e4c4c85917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:50:39.673026 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.672988    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b706ed-062e-40a3-8044-bafc0c131f07-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:39.673026 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.673023    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nmrt\" (UniqueName: \"kubernetes.io/projected/88b706ed-062e-40a3-8044-bafc0c131f07-kube-api-access-2nmrt\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:39.673232 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.673039    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b08b03b9-4124-4294-bab1-29e4c4c85917-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:39.675037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.675002    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08b03b9-4124-4294-bab1-29e4c4c85917-kube-api-access-rq9t8" pod "b08b03b9-4124-4294-bab1-29e4c4c85917" (UID: "b08b03b9-4124-4294-bab1-29e4c4c85917"). InnerVolumeSpecName "kube-api-access-rq9t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:39.763222 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.753107    5112 scope.go:122] "RemoveContainer" containerID="66dccaa33192035c1168e82b9d5f0761f96af7fe1cc76218db395a2831c0b84c"
Nov 03 16:50:39.786220 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.778834    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9t8\" (UniqueName: \"kubernetes.io/projected/b08b03b9-4124-4294-bab1-29e4c4c85917-kube-api-access-rq9t8\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:39.954619 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.954588    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"]
Nov 03 16:50:39.965353 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:39.965324    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6ae690-5d7956d5c5-gw8bl"]
Nov 03 16:50:40.453629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:40.453591    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z" event={"ID":"b08b03b9-4124-4294-bab1-29e4c4c85917","Type":"ContainerDied","Data":"09507d84a5b71315c0c6880e4860e216a474e1524dc01a899d788cdf4b5abb6c"}
Nov 03 16:50:40.454049 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:40.453604    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"
Nov 03 16:50:40.552977 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:40.552943    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b706ed-062e-40a3-8044-bafc0c131f07" path="/var/lib/kubelet/pods/88b706ed-062e-40a3-8044-bafc0c131f07/volumes"
Nov 03 16:50:40.676953 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:40.676913    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"]
Nov 03 16:50:40.682821 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:40.682787    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6ae690-bc76bf5b7-j4p4z"]
Nov 03 16:50:42.552603 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:42.552562    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08b03b9-4124-4294-bab1-29e4c4c85917" path="/var/lib/kubelet/pods/b08b03b9-4124-4294-bab1-29e4c4c85917/volumes"
Nov 03 16:50:46.016442 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.016405    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"]
Nov 03 16:50:46.016864 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.016678    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb" podUID="ae03bf56-8bee-4c18-94e6-351b4dedba85" containerName="maas-api" containerID="cri-o://5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0" gracePeriod=30
Nov 03 16:50:46.475338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.475306    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:50:46.500109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.500072    5112 generic.go:356] "Generic (PLEG): container finished" podID="ae03bf56-8bee-4c18-94e6-351b4dedba85" containerID="5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0" exitCode=0
Nov 03 16:50:46.500109 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.500114    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"
Nov 03 16:50:46.500340 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.500148    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb" event={"ID":"ae03bf56-8bee-4c18-94e6-351b4dedba85","Type":"ContainerDied","Data":"5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0"}
Nov 03 16:50:46.500340 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.500229    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb" event={"ID":"ae03bf56-8bee-4c18-94e6-351b4dedba85","Type":"ContainerDied","Data":"6b659b92eb91ee0c28ffb9ef22c74a0d1b93cc8e1ff2bda45c69520a8f10a388"}
Nov 03 16:50:46.500340 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.500255    5112 scope.go:122] "RemoveContainer" containerID="5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0"
Nov 03 16:50:46.510577 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.510553    5112 scope.go:122] "RemoveContainer" containerID="5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0"
Nov 03 16:50:46.510831 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:50:46.510814    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0\": container with ID starting with 5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0 not found: ID does not exist" containerID="5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0"
Nov 03 16:50:46.510898 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.510841    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0"} err="failed to get container status \"5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0\": rpc error: code = NotFound desc = could not find container \"5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0\": container with ID starting with 5ac81fa683e656ceeb34571570d4bbcd99c17eaeea8841bfefcd2cd8dce70fb0 not found: ID does not exist"
Nov 03 16:50:46.591602 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.591517    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae03bf56-8bee-4c18-94e6-351b4dedba85-kube-api-access-tpc9z\" (UniqueName: \"kubernetes.io/projected/ae03bf56-8bee-4c18-94e6-351b4dedba85-kube-api-access-tpc9z\") pod \"ae03bf56-8bee-4c18-94e6-351b4dedba85\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") "
Nov 03 16:50:46.591872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.591674    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae03bf56-8bee-4c18-94e6-351b4dedba85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae03bf56-8bee-4c18-94e6-351b4dedba85-openshift-service-ca\") pod \"ae03bf56-8bee-4c18-94e6-351b4dedba85\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") "
Nov 03 16:50:46.591872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.591747    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls\") pod \"ae03bf56-8bee-4c18-94e6-351b4dedba85\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") "
Nov 03 16:50:46.591872 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.591815    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-metrics-certs\") pod \"ae03bf56-8bee-4c18-94e6-351b4dedba85\" (UID: \"ae03bf56-8bee-4c18-94e6-351b4dedba85\") "
Nov 03 16:50:46.592837 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.592433    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae03bf56-8bee-4c18-94e6-351b4dedba85-openshift-service-ca" pod "ae03bf56-8bee-4c18-94e6-351b4dedba85" (UID: "ae03bf56-8bee-4c18-94e6-351b4dedba85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:50:46.600680 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.600643    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls" pod "ae03bf56-8bee-4c18-94e6-351b4dedba85" (UID: "ae03bf56-8bee-4c18-94e6-351b4dedba85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:50:46.600680 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.600664    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-metrics-certs" pod "ae03bf56-8bee-4c18-94e6-351b4dedba85" (UID: "ae03bf56-8bee-4c18-94e6-351b4dedba85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:50:46.600856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.600838    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae03bf56-8bee-4c18-94e6-351b4dedba85-kube-api-access-tpc9z" pod "ae03bf56-8bee-4c18-94e6-351b4dedba85" (UID: "ae03bf56-8bee-4c18-94e6-351b4dedba85"). InnerVolumeSpecName "kube-api-access-tpc9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:50:46.693597 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.693557    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpc9z\" (UniqueName: \"kubernetes.io/projected/ae03bf56-8bee-4c18-94e6-351b4dedba85-kube-api-access-tpc9z\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:46.693597 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.693587    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae03bf56-8bee-4c18-94e6-351b4dedba85-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:46.693597 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.693600    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:46.693869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.693614    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae03bf56-8bee-4c18-94e6-351b4dedba85-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:50:46.920365 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.920324    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"]
Nov 03 16:50:46.928933 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:46.928891    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-547a38-864568645c-x76tb"]
Nov 03 16:50:48.550416 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:48.550381    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae03bf56-8bee-4c18-94e6-351b4dedba85" path="/var/lib/kubelet/pods/ae03bf56-8bee-4c18-94e6-351b4dedba85/volumes"
Nov 03 16:50:57.269784 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:57.269749    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"]
Nov 03 16:50:57.270164 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:50:57.269959    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy" containerID="cri-o://84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f" gracePeriod=30
Nov 03 16:51:02.568860 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.568830    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:51:02.611856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.611818    5112 generic.go:356] "Generic (PLEG): container finished" podID="c319dc61-43c9-4409-9d22-b81507075236" containerID="84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f" exitCode=0
Nov 03 16:51:02.612019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.611873    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"
Nov 03 16:51:02.612019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.611900    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" event={"ID":"c319dc61-43c9-4409-9d22-b81507075236","Type":"ContainerDied","Data":"84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f"}
Nov 03 16:51:02.612019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.611956    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf" event={"ID":"c319dc61-43c9-4409-9d22-b81507075236","Type":"ContainerDied","Data":"536146626ec5d19a475b9822603644a1165c8c087e563bc334317714aa804a9c"}
Nov 03 16:51:02.612019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.611981    5112 scope.go:122] "RemoveContainer" containerID="84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f"
Nov 03 16:51:02.622106 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.622083    5112 scope.go:122] "RemoveContainer" containerID="84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f"
Nov 03 16:51:02.622443 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:02.622415    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f\": container with ID starting with 84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f not found: ID does not exist" containerID="84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f"
Nov 03 16:51:02.622489 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.622445    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f"} err="failed to get container status \"84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f\": rpc error: code = NotFound desc = could not find container \"84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f\": container with ID starting with 84e3fdbb2cc2058646f8e50ebaee364caee390a6f79121eaa0da89b0d00b608f not found: ID does not exist"
Nov 03 16:51:02.694856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.694819    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-data\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.694876    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-kube-api-access-zcdxv\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-kube-api-access-zcdxv\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.694940    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-istio-token\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-istio-token\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.694960    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c319dc61-43c9-4409-9d22-b81507075236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319dc61-43c9-4409-9d22-b81507075236-istio-podinfo\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695055 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695021    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-certs\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695070    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-socket\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695116    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-credential-socket\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695138    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-data" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:02.695346 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695283    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-envoy\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.695585 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695377    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-certs" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:02.695585 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695422    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-credential-socket" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:02.695585 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695434    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-socket" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:02.695800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.695777    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c319dc61-43c9-4409-9d22-b81507075236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319dc61-43c9-4409-9d22-b81507075236-istiod-ca-cert\") pod \"c319dc61-43c9-4409-9d22-b81507075236\" (UID: \"c319dc61-43c9-4409-9d22-b81507075236\") "
Nov 03 16:51:02.696273 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.696251    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c319dc61-43c9-4409-9d22-b81507075236-istiod-ca-cert" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:51:02.696447 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.696431    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c319dc61-43c9-4409-9d22-b81507075236-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.696520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.696454    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.696520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.696468    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.696520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.696482    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.696520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.696495    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.697563 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.697533    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-kube-api-access-zcdxv" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "kube-api-access-zcdxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:02.697658 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.697570    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-istio-token" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:02.697706 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.697687    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-envoy" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:02.697752 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.697709    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c319dc61-43c9-4409-9d22-b81507075236-istio-podinfo" pod "c319dc61-43c9-4409-9d22-b81507075236" (UID: "c319dc61-43c9-4409-9d22-b81507075236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:51:02.798545 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.798500    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcdxv\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-kube-api-access-zcdxv\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.798545 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.798543    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c319dc61-43c9-4409-9d22-b81507075236-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.798750 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.798561    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c319dc61-43c9-4409-9d22-b81507075236-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:02.798750 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:02.798575    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c319dc61-43c9-4409-9d22-b81507075236-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:03.055211 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:03.055156    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"]
Nov 03 16:51:03.063758 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:03.063720    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-547a38-openshift-default-cddffb669-jc5qf"]
Nov 03 16:51:04.550254 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:04.550193    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c319dc61-43c9-4409-9d22-b81507075236" path="/var/lib/kubelet/pods/c319dc61-43c9-4409-9d22-b81507075236/volumes"
Nov 03 16:51:07.885467 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:07.885425    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:51:08.094964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.094924    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"]
Nov 03 16:51:08.095242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.095213    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd" podUID="72538d58-0638-40b4-8c19-fe9a18259d2a" containerName="maas-api" containerID="cri-o://87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9" gracePeriod=30
Nov 03 16:51:08.491701 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.488094    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:51:08.593920 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.593873    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls\") pod \"72538d58-0638-40b4-8c19-fe9a18259d2a\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") "
Nov 03 16:51:08.594129 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.594045    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72538d58-0638-40b4-8c19-fe9a18259d2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72538d58-0638-40b4-8c19-fe9a18259d2a-openshift-service-ca\") pod \"72538d58-0638-40b4-8c19-fe9a18259d2a\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") "
Nov 03 16:51:08.594129 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.594095    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72538d58-0638-40b4-8c19-fe9a18259d2a-kube-api-access-k6frp\" (UniqueName: \"kubernetes.io/projected/72538d58-0638-40b4-8c19-fe9a18259d2a-kube-api-access-k6frp\") pod \"72538d58-0638-40b4-8c19-fe9a18259d2a\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") "
Nov 03 16:51:08.594415 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.594178    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-metrics-certs\") pod \"72538d58-0638-40b4-8c19-fe9a18259d2a\" (UID: \"72538d58-0638-40b4-8c19-fe9a18259d2a\") "
Nov 03 16:51:08.594476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.594422    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72538d58-0638-40b4-8c19-fe9a18259d2a-openshift-service-ca" pod "72538d58-0638-40b4-8c19-fe9a18259d2a" (UID: "72538d58-0638-40b4-8c19-fe9a18259d2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:51:08.595907 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.595876    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72538d58-0638-40b4-8c19-fe9a18259d2a-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:08.596401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.596378    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-metrics-certs" pod "72538d58-0638-40b4-8c19-fe9a18259d2a" (UID: "72538d58-0638-40b4-8c19-fe9a18259d2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:51:08.597027 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.597000    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72538d58-0638-40b4-8c19-fe9a18259d2a-kube-api-access-k6frp" pod "72538d58-0638-40b4-8c19-fe9a18259d2a" (UID: "72538d58-0638-40b4-8c19-fe9a18259d2a"). InnerVolumeSpecName "kube-api-access-k6frp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:08.597118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.597075    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls" pod "72538d58-0638-40b4-8c19-fe9a18259d2a" (UID: "72538d58-0638-40b4-8c19-fe9a18259d2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:51:08.663660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.658226    5112 generic.go:356] "Generic (PLEG): container finished" podID="72538d58-0638-40b4-8c19-fe9a18259d2a" containerID="87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9" exitCode=0
Nov 03 16:51:08.663660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.658304    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"
Nov 03 16:51:08.663660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.658329    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd" event={"ID":"72538d58-0638-40b4-8c19-fe9a18259d2a","Type":"ContainerDied","Data":"87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9"}
Nov 03 16:51:08.663660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.658401    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd" event={"ID":"72538d58-0638-40b4-8c19-fe9a18259d2a","Type":"ContainerDied","Data":"0d23c9c0b7babb89ff65d5766544fdd06f68401fde6687a15e8bce49f187540a"}
Nov 03 16:51:08.663660 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.658432    5112 scope.go:122] "RemoveContainer" containerID="87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9"
Nov 03 16:51:08.675256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.674244    5112 scope.go:122] "RemoveContainer" containerID="87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9"
Nov 03 16:51:08.675256 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:08.674634    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9\": container with ID starting with 87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9 not found: ID does not exist" containerID="87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9"
Nov 03 16:51:08.675256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.674669    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9"} err="failed to get container status \"87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9\": rpc error: code = NotFound desc = could not find container \"87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9\": container with ID starting with 87d77987c63686e49f8f092628abaafbbf99fea3441a50f37b02a83b656868c9 not found: ID does not exist"
Nov 03 16:51:08.696870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.696834    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6frp\" (UniqueName: \"kubernetes.io/projected/72538d58-0638-40b4-8c19-fe9a18259d2a-kube-api-access-k6frp\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:08.696870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.696867    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:08.697080 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.696881    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72538d58-0638-40b4-8c19-fe9a18259d2a-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:08.829857 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.829760    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"]
Nov 03 16:51:08.834734 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:08.834701    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-192564-84f4757c87-z6lmd"]
Nov 03 16:51:10.551805 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:10.551762    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72538d58-0638-40b4-8c19-fe9a18259d2a" path="/var/lib/kubelet/pods/72538d58-0638-40b4-8c19-fe9a18259d2a/volumes"
Nov 03 16:51:19.683046 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:19.682958    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"]
Nov 03 16:51:19.683471 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:19.683215    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" containerID="cri-o://c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7" gracePeriod=30
Nov 03 16:51:22.472918 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:22.472874    5112 patch_prober.go:29] interesting pod/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:51:22.473316 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:22.472945    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:51:24.955306 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:24.955253    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:51:25.047857 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.047830    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-certs\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.047984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.047925    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-kube-api-access-6p9z5\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-kube-api-access-6p9z5\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.047984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.047974    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-envoy\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.048086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048015    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-istio-token\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-istio-token\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.048086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048048    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-data\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.048086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048065    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-certs" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:25.048086 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048080    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-socket\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.048274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048112    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8887665-7239-480d-b6d1-c0352e1edce5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8887665-7239-480d-b6d1-c0352e1edce5-istiod-ca-cert\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.048274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048155    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-credential-socket\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.048274 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048254    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8887665-7239-480d-b6d1-c0352e1edce5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8887665-7239-480d-b6d1-c0352e1edce5-istio-podinfo\") pod \"d8887665-7239-480d-b6d1-c0352e1edce5\" (UID: \"d8887665-7239-480d-b6d1-c0352e1edce5\") "
Nov 03 16:51:25.048387 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048361    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-data" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:25.048698 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048673    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-socket" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:25.048801 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048690    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-credential-socket" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:25.048801 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048776    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.048801 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048794    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.048964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048804    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.048964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048813    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.048964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.048808    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8887665-7239-480d-b6d1-c0352e1edce5-istiod-ca-cert" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:51:25.050514 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.050479    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-kube-api-access-6p9z5" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "kube-api-access-6p9z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:25.050782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.050744    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-istio-token" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:25.050882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.050851    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-envoy" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:25.051294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.051269    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8887665-7239-480d-b6d1-c0352e1edce5-istio-podinfo" pod "d8887665-7239-480d-b6d1-c0352e1edce5" (UID: "d8887665-7239-480d-b6d1-c0352e1edce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:51:25.150466 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.150439    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8887665-7239-480d-b6d1-c0352e1edce5-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.150582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.150471    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.150582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.150488    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8887665-7239-480d-b6d1-c0352e1edce5-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.150582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.150501    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8887665-7239-480d-b6d1-c0352e1edce5-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.150582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.150513    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p9z5\" (UniqueName: \"kubernetes.io/projected/d8887665-7239-480d-b6d1-c0352e1edce5-kube-api-access-6p9z5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:25.778437 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.778404    5112 generic.go:356] "Generic (PLEG): container finished" podID="d8887665-7239-480d-b6d1-c0352e1edce5" containerID="c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7" exitCode=0
Nov 03 16:51:25.778613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.778454    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"
Nov 03 16:51:25.778613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.778490    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" event={"ID":"d8887665-7239-480d-b6d1-c0352e1edce5","Type":"ContainerDied","Data":"c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7"}
Nov 03 16:51:25.778613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.778547    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw" event={"ID":"d8887665-7239-480d-b6d1-c0352e1edce5","Type":"ContainerDied","Data":"b2572b0fb27a67e412dce04fc102d8e2eff54289e79d950b9a1c94b0f2137cbe"}
Nov 03 16:51:25.778613 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.778576    5112 scope.go:122] "RemoveContainer" containerID="c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7"
Nov 03 16:51:25.788965 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.788947    5112 scope.go:122] "RemoveContainer" containerID="c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7"
Nov 03 16:51:25.789261 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:25.789240    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7\": container with ID starting with c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7 not found: ID does not exist" containerID="c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7"
Nov 03 16:51:25.789318 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.789270    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7"} err="failed to get container status \"c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7\": rpc error: code = NotFound desc = could not find container \"c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7\": container with ID starting with c4b6ddb4be250758c02ba021ae7bcd51e722ef9431e5897511e6e993c84392b7 not found: ID does not exist"
Nov 03 16:51:25.989262 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:25.989229    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"]
Nov 03 16:51:26.034256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:26.034164    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-192564-openshift-default-6d8f48dd8f-7kdqw"]
Nov 03 16:51:26.549993 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:26.549963    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" path="/var/lib/kubelet/pods/d8887665-7239-480d-b6d1-c0352e1edce5/volumes"
Nov 03 16:51:29.554415 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.554382    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"]
Nov 03 16:51:29.555184 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555162    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f710392e-71fa-4bcf-af87-96ec3ea904c3" containerName="payload-processing"
Nov 03 16:51:29.555184 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555185    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f710392e-71fa-4bcf-af87-96ec3ea904c3" containerName="payload-processing"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555212    5112 cpu_manager.go:414] "containerMap: removing container" podUID="c3d3d850-986d-4dbf-9854-a8c647328cb8" containerName="payload-pre-processing"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555218    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d3d850-986d-4dbf-9854-a8c647328cb8" containerName="payload-pre-processing"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555230    5112 cpu_manager.go:414] "containerMap: removing container" podUID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" containerName="payload-pre-processing"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555236    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" containerName="payload-pre-processing"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555251    5112 cpu_manager.go:414] "containerMap: removing container" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555256    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555262    5112 cpu_manager.go:414] "containerMap: removing container" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555267    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555273    5112 cpu_manager.go:414] "containerMap: removing container" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555278    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555286    5112 cpu_manager.go:414] "containerMap: removing container" podUID="6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555291    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555300    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b08b03b9-4124-4294-bab1-29e4c4c85917" containerName="payload-pre-processing"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555305    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08b03b9-4124-4294-bab1-29e4c4c85917" containerName="payload-pre-processing"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555313    5112 cpu_manager.go:414] "containerMap: removing container" podUID="53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555318    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555326    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a00cbb13-70ee-40a0-a1f8-f4982b375909" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555330    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00cbb13-70ee-40a0-a1f8-f4982b375909" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555338    5112 cpu_manager.go:414] "containerMap: removing container" podUID="72538d58-0638-40b4-8c19-fe9a18259d2a" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555343    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72538d58-0638-40b4-8c19-fe9a18259d2a" containerName="maas-api"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555354    5112 cpu_manager.go:414] "containerMap: removing container" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main"
Nov 03 16:51:29.555351 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555361    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555373    5112 cpu_manager.go:414] "containerMap: removing container" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555379    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555385    5112 cpu_manager.go:414] "containerMap: removing container" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555390    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555397    5112 cpu_manager.go:414] "containerMap: removing container" podUID="8677da00-1571-4dbf-ad73-6432017e538e" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555401    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8677da00-1571-4dbf-ad73-6432017e538e" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555408    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555412    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555431    5112 cpu_manager.go:414] "containerMap: removing container" podUID="59f795ab-eafb-46bd-969d-551771cdbd7a" containerName="payload-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555435    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f795ab-eafb-46bd-969d-551771cdbd7a" containerName="payload-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555445    5112 cpu_manager.go:414] "containerMap: removing container" podUID="88b706ed-062e-40a3-8044-bafc0c131f07" containerName="payload-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555449    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b706ed-062e-40a3-8044-bafc0c131f07" containerName="payload-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555455    5112 cpu_manager.go:414] "containerMap: removing container" podUID="ae03bf56-8bee-4c18-94e6-351b4dedba85" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555459    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae03bf56-8bee-4c18-94e6-351b4dedba85" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555468    5112 cpu_manager.go:414] "containerMap: removing container" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555472    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555481    5112 cpu_manager.go:414] "containerMap: removing container" podUID="c0866a99-1153-465a-90d6-06a647a8fda6" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555493    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0866a99-1153-465a-90d6-06a647a8fda6" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555502    5112 cpu_manager.go:414] "containerMap: removing container" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555506    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555512    5112 cpu_manager.go:414] "containerMap: removing container" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555517    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555527    5112 cpu_manager.go:414] "containerMap: removing container" podUID="07a6ee17-7736-44a8-8a55-9ad954129b8b" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555531    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a6ee17-7736-44a8-8a55-9ad954129b8b" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555539    5112 cpu_manager.go:414] "containerMap: removing container" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555543    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555551    5112 cpu_manager.go:414] "containerMap: removing container" podUID="d37bc61b-61dc-4b3b-94a3-bbc54babfca8" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555556    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37bc61b-61dc-4b3b-94a3-bbc54babfca8" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555634    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16acfc1-e819-4a7c-b026-c7ffde68da12" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555645    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f795ab-eafb-46bd-969d-551771cdbd7a" containerName="payload-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555651    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b706ed-062e-40a3-8044-bafc0c131f07" containerName="payload-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555658    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae03bf56-8bee-4c18-94e6-351b4dedba85" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555666    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8887665-7239-480d-b6d1-c0352e1edce5" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555672    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0866a99-1153-465a-90d6-06a647a8fda6" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555677    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d8b397-8b41-42c3-8bdf-8ef23ee8663d" containerName="main"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555682    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="681ca214-8021-47b1-b94f-73e47fea2010" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555689    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a6ee17-7736-44a8-8a55-9ad954129b8b" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555696    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="c319dc61-43c9-4409-9d22-b81507075236" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555701    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37bc61b-61dc-4b3b-94a3-bbc54babfca8" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555706    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="f710392e-71fa-4bcf-af87-96ec3ea904c3" containerName="payload-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555712    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d3d850-986d-4dbf-9854-a8c647328cb8" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555718    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cb2a2f-e7e9-4ec2-90b3-4c39bec86c08" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555724    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="1474dd53-f607-4a36-880a-670a21b9a1d8" containerName="main"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555729    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="348e1e8e-9cc9-4c31-9097-b766fe181877" containerName="main"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555734    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe61a3c-41f2-4688-9886-3cc710efb308" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555740    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="6835a522-d79e-4a1c-8dbf-8a6d04dc28e7" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555745    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08b03b9-4124-4294-bab1-29e4c4c85917" containerName="payload-pre-processing"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555751    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f4793c-bc75-4bab-8ae9-40f1c7fa9ae6" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555757    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00cbb13-70ee-40a0-a1f8-f4982b375909" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555763    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="72538d58-0638-40b4-8c19-fe9a18259d2a" containerName="maas-api"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555769    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfebe23a-01a1-4b27-ad8d-2497166eef3a" containerName="main"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555774    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0f5855-bc6a-4677-93ce-dd659bf590d7" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555779    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca07c6b-d01a-403d-81d6-2dbebcbd8c13" containerName="istio-proxy"
Nov 03 16:51:29.556037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.555785    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="8677da00-1571-4dbf-ad73-6432017e538e" containerName="maas-api"
Nov 03 16:51:29.561154 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.561135    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.607479 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.607450    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.607628 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.607518    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-metrics-certs\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.607628 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.607613    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.607749 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.607733    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmf2\" (UniqueName: \"kubernetes.io/projected/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-kube-api-access-swmf2\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.615112 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.615085    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"]
Nov 03 16:51:29.658294 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.658263    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ddf492-serving-cert\""
Nov 03 16:51:29.710225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.709561    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.710225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.709621    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-metrics-certs\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.710225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.709658    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.710225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.709742    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmf2\" (UniqueName: \"kubernetes.io/projected/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-kube-api-access-swmf2\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.710574 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.710388    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.710574 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:29.710496    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-serving-cert: secret "maas-api-e2e-inf-a-ddf492-serving-cert" not found
Nov 03 16:51:29.710574 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:29.710559    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls podName:8648b34b-c21b-49e6-b1d9-4b12b8df2c24 nodeName:}" failed. No retries permitted until 2026-11-03 16:51:30.210538204 +0000 UTC m=+2451.789842052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls") pod "maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz" (UID: "8648b34b-c21b-49e6-b1d9-4b12b8df2c24") : secret "maas-api-e2e-inf-a-ddf492-serving-cert" not found
Nov 03 16:51:29.712902 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.712878    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-metrics-certs\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:29.779064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:29.779032    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmf2\" (UniqueName: \"kubernetes.io/projected/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-kube-api-access-swmf2\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:30.218251 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:30.218184    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:30.220665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:30.220637    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:30.475808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:30.475718    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:30.852688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:30.852657    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"]
Nov 03 16:51:30.854415 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:51:30.854388    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8648b34b_c21b_49e6_b1d9_4b12b8df2c24.slice/crio-676fbd0de0ac76fda758bc34eb3a44b14bca06f6ab9e65f935415fc6ed040767 WatchSource:0}: Error finding container 676fbd0de0ac76fda758bc34eb3a44b14bca06f6ab9e65f935415fc6ed040767: Status 404 returned error can't find the container with id 676fbd0de0ac76fda758bc34eb3a44b14bca06f6ab9e65f935415fc6ed040767
Nov 03 16:51:30.856248 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:30.856228    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 16:51:31.840552 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:31.840513    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz" event={"ID":"8648b34b-c21b-49e6-b1d9-4b12b8df2c24","Type":"ContainerStarted","Data":"8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8"}
Nov 03 16:51:31.840734 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:31.840605    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz" event={"ID":"8648b34b-c21b-49e6-b1d9-4b12b8df2c24","Type":"ContainerStarted","Data":"676fbd0de0ac76fda758bc34eb3a44b14bca06f6ab9e65f935415fc6ed040767"}
Nov 03 16:51:31.840734 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:31.840631    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:31.968367 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:31.968251    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz" podStartSLOduration=2.172585066 podStartE2EDuration="2.968237432s" totalImagesPullingTime="795.652366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:31.965835538 +0000 UTC m=+2453.545139412" watchObservedRunningTime="2026-11-03 16:51:31.968237432 +0000 UTC m=+2453.547541286"
Nov 03 16:51:37.853317 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:37.853285    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 16:51:37.885019 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:37.884983    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:51:39.924869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:39.924840    5112 scope.go:122] "RemoveContainer" containerID="95877921aa1ad7a2d3d3174a1b9822a2b84608fdd5cd8dc68483998ed6fce3f7"
Nov 03 16:51:43.389507 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.389470    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"]
Nov 03 16:51:43.390100 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.389685    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2" podUID="8fc2a681-7950-4b44-89d2-18f34e8ae930" containerName="maas-api" containerID="cri-o://3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb" gracePeriod=30
Nov 03 16:51:43.720551 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.720522    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:51:43.831260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.831184    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-metrics-certs\") pod \"8fc2a681-7950-4b44-89d2-18f34e8ae930\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") "
Nov 03 16:51:43.831665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.831400    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc2a681-7950-4b44-89d2-18f34e8ae930-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2a681-7950-4b44-89d2-18f34e8ae930-openshift-service-ca\") pod \"8fc2a681-7950-4b44-89d2-18f34e8ae930\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") "
Nov 03 16:51:43.831665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.831421    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls\") pod \"8fc2a681-7950-4b44-89d2-18f34e8ae930\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") "
Nov 03 16:51:43.831665 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.831527    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc2a681-7950-4b44-89d2-18f34e8ae930-kube-api-access-6vdlq\" (UniqueName: \"kubernetes.io/projected/8fc2a681-7950-4b44-89d2-18f34e8ae930-kube-api-access-6vdlq\") pod \"8fc2a681-7950-4b44-89d2-18f34e8ae930\" (UID: \"8fc2a681-7950-4b44-89d2-18f34e8ae930\") "
Nov 03 16:51:43.831847 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.831759    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc2a681-7950-4b44-89d2-18f34e8ae930-openshift-service-ca" pod "8fc2a681-7950-4b44-89d2-18f34e8ae930" (UID: "8fc2a681-7950-4b44-89d2-18f34e8ae930"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:51:43.831994 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.831971    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2a681-7950-4b44-89d2-18f34e8ae930-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:43.833489 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.833454    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-metrics-certs" pod "8fc2a681-7950-4b44-89d2-18f34e8ae930" (UID: "8fc2a681-7950-4b44-89d2-18f34e8ae930"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:51:43.833672 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.833650    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc2a681-7950-4b44-89d2-18f34e8ae930-kube-api-access-6vdlq" pod "8fc2a681-7950-4b44-89d2-18f34e8ae930" (UID: "8fc2a681-7950-4b44-89d2-18f34e8ae930"). InnerVolumeSpecName "kube-api-access-6vdlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:43.833742 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.833716    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls" pod "8fc2a681-7950-4b44-89d2-18f34e8ae930" (UID: "8fc2a681-7950-4b44-89d2-18f34e8ae930"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:51:43.926787 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.926747    5112 generic.go:356] "Generic (PLEG): container finished" podID="8fc2a681-7950-4b44-89d2-18f34e8ae930" containerID="3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb" exitCode=0
Nov 03 16:51:43.926998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.926795    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"
Nov 03 16:51:43.926998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.926832    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2" event={"ID":"8fc2a681-7950-4b44-89d2-18f34e8ae930","Type":"ContainerDied","Data":"3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb"}
Nov 03 16:51:43.926998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.926892    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2" event={"ID":"8fc2a681-7950-4b44-89d2-18f34e8ae930","Type":"ContainerDied","Data":"f27ace0ed2c297dccdc928da03d2311498cdacfbc30afaffdaafc8ea5e7309eb"}
Nov 03 16:51:43.926998 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.926921    5112 scope.go:122] "RemoveContainer" containerID="3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb"
Nov 03 16:51:43.932819 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.932794    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:43.932819 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.932818    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc2a681-7950-4b44-89d2-18f34e8ae930-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:43.933021 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.932831    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdlq\" (UniqueName: \"kubernetes.io/projected/8fc2a681-7950-4b44-89d2-18f34e8ae930-kube-api-access-6vdlq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:43.938284 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.938257    5112 scope.go:122] "RemoveContainer" containerID="3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb"
Nov 03 16:51:43.938592 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:43.938575    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb\": container with ID starting with 3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb not found: ID does not exist" containerID="3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb"
Nov 03 16:51:43.938650 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:43.938602    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb"} err="failed to get container status \"3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb\": rpc error: code = NotFound desc = could not find container \"3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb\": container with ID starting with 3806819fd95b43109c960ca57bd5bef62a03c5b65a50ba585fad8b2e829411fb not found: ID does not exist"
Nov 03 16:51:44.089610 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:44.089496    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"]
Nov 03 16:51:44.095309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:44.095275    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6ae690-849f7c87-s9sb2"]
Nov 03 16:51:44.550461 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:44.550431    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc2a681-7950-4b44-89d2-18f34e8ae930" path="/var/lib/kubelet/pods/8fc2a681-7950-4b44-89d2-18f34e8ae930/volumes"
Nov 03 16:51:53.431888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:53.431844    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"]
Nov 03 16:51:53.432354 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:53.432052    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" containerID="cri-o://87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c" gracePeriod=30
Nov 03 16:51:53.620844 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:53.620802    5112 patch_prober.go:29] interesting pod/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:51:53.621016 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:53.620880    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:51:58.687882 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.687853    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:51:58.828856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.828749    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-socket\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.828856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.828800    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-data\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.828856 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.828845    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-credential-socket\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.829242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.828898    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-envoy\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.829242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829049    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-certs\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.829242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829053    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-data" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:58.829242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829083    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-token\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-token\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.829242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829093    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-socket" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:58.829242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829158    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-kube-api-access-cqgjn\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-kube-api-access-cqgjn\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.829242 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829226    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istiod-ca-cert\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.829651 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829246    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-credential-socket" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:58.829651 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829285    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-podinfo\") pod \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\" (UID: \"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec\") "
Nov 03 16:51:58.829651 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829333    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-certs" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:58.829822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829793    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.829822 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829814    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.829934 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829830    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.829934 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.829845    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.830841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.830170    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istiod-ca-cert" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:51:58.832059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.831954    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-envoy" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:51:58.832059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.832018    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-kube-api-access-cqgjn" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "kube-api-access-cqgjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:58.833467 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.833396    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-token" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:51:58.833832 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.833808    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-podinfo" pod "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" (UID: "eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:51:58.931543 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.931507    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.931543 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.931543    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.931737 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.931556    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.931737 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.931566    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqgjn\" (UniqueName: \"kubernetes.io/projected/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-kube-api-access-cqgjn\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:58.931737 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:58.931575    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:51:59.046415 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.046376    5112 generic.go:356] "Generic (PLEG): container finished" podID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerID="87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c" exitCode=0
Nov 03 16:51:59.046592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.046425    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"
Nov 03 16:51:59.046592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.046466    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" event={"ID":"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec","Type":"ContainerDied","Data":"87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c"}
Nov 03 16:51:59.046592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.046523    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6" event={"ID":"eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec","Type":"ContainerDied","Data":"0e8be5ad1c84535f1ff2aebdbd62441151070ba76aff49f1e6daef42809cf081"}
Nov 03 16:51:59.046592 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.046546    5112 scope.go:122] "RemoveContainer" containerID="87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c"
Nov 03 16:51:59.059654 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.059602    5112 scope.go:122] "RemoveContainer" containerID="87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c"
Nov 03 16:51:59.061405 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:51:59.059963    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c\": container with ID starting with 87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c not found: ID does not exist" containerID="87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c"
Nov 03 16:51:59.061405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.060009    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c"} err="failed to get container status \"87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c\": rpc error: code = NotFound desc = could not find container \"87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c\": container with ID starting with 87d2fb049817f0b182427d00dd4b18291ef86f7dc1400cff87360e9526e7524c not found: ID does not exist"
Nov 03 16:51:59.212265 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.212225    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"]
Nov 03 16:51:59.223405 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:51:59.223369    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6ae690-openshift-default-6799b84845-xrvr6"]
Nov 03 16:52:00.549841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:00.549800    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" path="/var/lib/kubelet/pods/eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec/volumes"
Nov 03 16:52:05.536105 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.536066    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"]
Nov 03 16:52:05.536731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.536704    5112 cpu_manager.go:414] "containerMap: removing container" podUID="8fc2a681-7950-4b44-89d2-18f34e8ae930" containerName="maas-api"
Nov 03 16:52:05.536731 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.536727    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc2a681-7950-4b44-89d2-18f34e8ae930" containerName="maas-api"
Nov 03 16:52:05.536942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.536753    5112 cpu_manager.go:414] "containerMap: removing container" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy"
Nov 03 16:52:05.536942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.536758    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy"
Nov 03 16:52:05.536942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.536885    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc2a681-7950-4b44-89d2-18f34e8ae930" containerName="maas-api"
Nov 03 16:52:05.536942 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.536899    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb9dd54f-d7e6-4a97-9ffc-45326e19d6ec" containerName="istio-proxy"
Nov 03 16:52:05.540468 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.540448    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.598045 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.598009    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"]
Nov 03 16:52:05.640518 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.640483    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-244c65-serving-cert\""
Nov 03 16:52:05.710836 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.710798    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-metrics-certs\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.711019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.710853    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvmf\" (UniqueName: \"kubernetes.io/projected/2144027f-280e-486c-a857-1efab523a47d-kube-api-access-kpvmf\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.711019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.710997    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-maas-api-tls\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.711108 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.711059    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2144027f-280e-486c-a857-1efab523a47d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.812774 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.812668    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-metrics-certs\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.812774 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.812736    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvmf\" (UniqueName: \"kubernetes.io/projected/2144027f-280e-486c-a857-1efab523a47d-kube-api-access-kpvmf\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.813016 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.812842    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-maas-api-tls\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.813016 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.812905    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2144027f-280e-486c-a857-1efab523a47d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.813819 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.813788    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2144027f-280e-486c-a857-1efab523a47d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.815944 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.815916    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-metrics-certs\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.816094 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.816071    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-maas-api-tls\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:05.871513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:05.871466    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvmf\" (UniqueName: \"kubernetes.io/projected/2144027f-280e-486c-a857-1efab523a47d-kube-api-access-kpvmf\") pod \"maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:06.156671 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:06.156634    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:06.540809 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:06.540779    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"]
Nov 03 16:52:06.541622 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:52:06.541591    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2144027f_280e_486c_a857_1efab523a47d.slice/crio-413c545effc3b9cb7cfe22634470af9cacb41d4d145516d8f210cedea065c1ef WatchSource:0}: Error finding container 413c545effc3b9cb7cfe22634470af9cacb41d4d145516d8f210cedea065c1ef: Status 404 returned error can't find the container with id 413c545effc3b9cb7cfe22634470af9cacb41d4d145516d8f210cedea065c1ef
Nov 03 16:52:07.106539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:07.106499    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj" event={"ID":"2144027f-280e-486c-a857-1efab523a47d","Type":"ContainerStarted","Data":"413c545effc3b9cb7cfe22634470af9cacb41d4d145516d8f210cedea065c1ef"}
Nov 03 16:52:07.884464 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:52:07.884354    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:52:08.114871 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:08.114827    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj" event={"ID":"2144027f-280e-486c-a857-1efab523a47d","Type":"ContainerStarted","Data":"224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254"}
Nov 03 16:52:08.115069 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:08.115008    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:08.229721 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:08.229668    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj" podStartSLOduration=2.341450396 podStartE2EDuration="3.229653471s" totalImagesPullingTime="888.203075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:08.229279433 +0000 UTC m=+2489.808583303" watchObservedRunningTime="2026-11-03 16:52:08.229653471 +0000 UTC m=+2489.808957326"
Nov 03 16:52:14.126309 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:52:14.126275    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:52:37.886670 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:52:37.886626    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:53:07.884750 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:53:07.884659    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:53:33.674505 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.674469    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"]
Nov 03 16:53:33.679246 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.679216    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.778722 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.778681    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-61be78-serving-cert\""
Nov 03 16:53:33.781841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.781809    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"]
Nov 03 16:53:33.799793 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.799750    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.799995 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.799905    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.799995 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.799965    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-metrics-certs\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.800137 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.800042    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9t9\" (UniqueName: \"kubernetes.io/projected/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-kube-api-access-2c9t9\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.901073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.901038    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.901306 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.901128    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.901306 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.901186    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-metrics-certs\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.901306 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:53:33.901215    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-serving-cert: secret "maas-api-e2e-inf-b-61be78-serving-cert" not found
Nov 03 16:53:33.901306 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:53:33.901294    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls podName:8d628a4f-6ead-4e79-b7cd-57a2806f2e21 nodeName:}" failed. No retries permitted until 2026-11-03 16:53:34.401274907 +0000 UTC m=+2575.980578740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls") pod "maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz" (UID: "8d628a4f-6ead-4e79-b7cd-57a2806f2e21") : secret "maas-api-e2e-inf-b-61be78-serving-cert" not found
Nov 03 16:53:33.901538 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.901331    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9t9\" (UniqueName: \"kubernetes.io/projected/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-kube-api-access-2c9t9\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.901984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.901957    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.904534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.904510    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-metrics-certs\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:33.964289 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:33.958898    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9t9\" (UniqueName: \"kubernetes.io/projected/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-kube-api-access-2c9t9\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:34.407581 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:34.407538    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:34.410059 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:34.410040    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls\") pod \"maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:34.593948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:34.593909    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:34.970678 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:34.970646    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"]
Nov 03 16:53:34.972098 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:53:34.972059    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d628a4f_6ead_4e79_b7cd_57a2806f2e21.slice/crio-6ac741f4e2b1d72b037c175ea9890966d1956c0ae02e56fdd9068c0b508a23ef WatchSource:0}: Error finding container 6ac741f4e2b1d72b037c175ea9890966d1956c0ae02e56fdd9068c0b508a23ef: Status 404 returned error can't find the container with id 6ac741f4e2b1d72b037c175ea9890966d1956c0ae02e56fdd9068c0b508a23ef
Nov 03 16:53:35.771524 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:35.771474    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz" event={"ID":"8d628a4f-6ead-4e79-b7cd-57a2806f2e21","Type":"ContainerStarted","Data":"6ac741f4e2b1d72b037c175ea9890966d1956c0ae02e56fdd9068c0b508a23ef"}
Nov 03 16:53:36.780419 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:36.780382    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz" event={"ID":"8d628a4f-6ead-4e79-b7cd-57a2806f2e21","Type":"ContainerStarted","Data":"a35f6ed4ee35f7253d9096ad276fad7e42ea5435b77ce74cf8651089eeccf950"}
Nov 03 16:53:36.780876 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:36.780446    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:36.895904 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:36.895838    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz" podStartSLOduration=2.990902959 podStartE2EDuration="3.895818279s" totalImagesPullingTime="904.91532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:36.894004052 +0000 UTC m=+2578.473307928" watchObservedRunningTime="2026-11-03 16:53:36.895818279 +0000 UTC m=+2578.475122136"
Nov 03 16:53:37.883516 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:53:37.883467    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:53:42.798657 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:42.798613    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 16:53:47.987448 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:47.987412    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"]
Nov 03 16:53:47.991524 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:47.991502    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.042982 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.042953    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-963a2f-serving-cert\""
Nov 03 16:53:48.080987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.080945    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d684-66ba-485c-a926-deaa20d41fd6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.080987 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.080988    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8x62\" (UniqueName: \"kubernetes.io/projected/acc9d684-66ba-485c-a926-deaa20d41fd6-kube-api-access-l8x62\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.081256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.081025    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-metrics-certs\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.081256 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.081152    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.092317 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.092288    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"]
Nov 03 16:53:48.182624 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.182583    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-metrics-certs\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.182831 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.182642    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.182831 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:53:48.182804    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-serving-cert: secret "maas-api-e2e-inf-b-963a2f-serving-cert" not found
Nov 03 16:53:48.182949 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.182847    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d684-66ba-485c-a926-deaa20d41fd6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.182949 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:53:48.182881    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls podName:acc9d684-66ba-485c-a926-deaa20d41fd6 nodeName:}" failed. No retries permitted until 2026-11-03 16:53:48.682865785 +0000 UTC m=+2590.262169619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls") pod "maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk" (UID: "acc9d684-66ba-485c-a926-deaa20d41fd6") : secret "maas-api-e2e-inf-b-963a2f-serving-cert" not found
Nov 03 16:53:48.182949 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.182924    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8x62\" (UniqueName: \"kubernetes.io/projected/acc9d684-66ba-485c-a926-deaa20d41fd6-kube-api-access-l8x62\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.183623 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.183598    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d684-66ba-485c-a926-deaa20d41fd6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.185690 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.185663    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-metrics-certs\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.239269 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.239181    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8x62\" (UniqueName: \"kubernetes.io/projected/acc9d684-66ba-485c-a926-deaa20d41fd6-kube-api-access-l8x62\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.688745 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.688703    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.691554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.691521    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls\") pod \"maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:48.906125 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:48.906087    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:49.297847 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:53:49.297808    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc9d684_66ba_485c_a926_deaa20d41fd6.slice/crio-aa5ea1da46f5f86bdb472ba052fa69c503adf10824c370365ddb073d67512b0f WatchSource:0}: Error finding container aa5ea1da46f5f86bdb472ba052fa69c503adf10824c370365ddb073d67512b0f: Status 404 returned error can't find the container with id aa5ea1da46f5f86bdb472ba052fa69c503adf10824c370365ddb073d67512b0f
Nov 03 16:53:49.299103 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:49.299075    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"]
Nov 03 16:53:49.883712 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:49.883675    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk" event={"ID":"acc9d684-66ba-485c-a926-deaa20d41fd6","Type":"ContainerStarted","Data":"aa5ea1da46f5f86bdb472ba052fa69c503adf10824c370365ddb073d67512b0f"}
Nov 03 16:53:50.894071 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:50.894028    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk" event={"ID":"acc9d684-66ba-485c-a926-deaa20d41fd6","Type":"ContainerStarted","Data":"9c7f2e5fe41391f88f5488ede56ea5bdfd656f5a59821196737ce6abd95a1413"}
Nov 03 16:53:50.894626 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:50.894092    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:53:51.009522 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:51.009462    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk" podStartSLOduration=3.132708647 podStartE2EDuration="4.009449122s" totalImagesPullingTime="876.740475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:51.008316907 +0000 UTC m=+2592.587620765" watchObservedRunningTime="2026-11-03 16:53:51.009449122 +0000 UTC m=+2592.588752977"
Nov 03 16:53:56.907754 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:53:56.907725    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 16:54:07.884392 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:54:07.884342    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:54:37.883968 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:54:37.883929    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:55:07.886239 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:55:07.886183    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:55:37.884245 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:55:37.884188    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:55:38.048977 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.048939    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"]
Nov 03 16:55:38.049189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.049125    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh" podUID="3fbe652f-e7ae-4938-bff9-c46e098caa35" containerName="discovery" containerID="cri-o://55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4" gracePeriod=30
Nov 03 16:55:38.341767 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.341736    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:55:38.434555 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.434465    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-kubeconfig\") pod \"3fbe652f-e7ae-4938-bff9-c46e098caa35\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") "
Nov 03 16:55:38.434555 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.434555    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-cacerts\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-cacerts\") pod \"3fbe652f-e7ae-4938-bff9-c46e098caa35\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") "
Nov 03 16:55:38.434789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.434591    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-ca-configmap\") pod \"3fbe652f-e7ae-4938-bff9-c46e098caa35\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") "
Nov 03 16:55:38.434789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.434640    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-token\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-token\") pod \"3fbe652f-e7ae-4938-bff9-c46e098caa35\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") "
Nov 03 16:55:38.434789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.434659    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-dns-cert\") pod \"3fbe652f-e7ae-4938-bff9-c46e098caa35\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") "
Nov 03 16:55:38.434789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.434736    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-kube-api-access-v6ngm\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-kube-api-access-v6ngm\") pod \"3fbe652f-e7ae-4938-bff9-c46e098caa35\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") "
Nov 03 16:55:38.435017 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.434988    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-ca-configmap" pod "3fbe652f-e7ae-4938-bff9-c46e098caa35" (UID: "3fbe652f-e7ae-4938-bff9-c46e098caa35"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:55:38.435179 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.435157    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbe652f-e7ae-4938-bff9-c46e098caa35-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbe652f-e7ae-4938-bff9-c46e098caa35-local-certs\") pod \"3fbe652f-e7ae-4938-bff9-c46e098caa35\" (UID: \"3fbe652f-e7ae-4938-bff9-c46e098caa35\") "
Nov 03 16:55:38.436189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.436170    5112 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-ca-configmap\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:55:38.437500 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.437474    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-cacerts" pod "3fbe652f-e7ae-4938-bff9-c46e098caa35" (UID: "3fbe652f-e7ae-4938-bff9-c46e098caa35"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:55:38.437634 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.437607    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-kubeconfig" pod "3fbe652f-e7ae-4938-bff9-c46e098caa35" (UID: "3fbe652f-e7ae-4938-bff9-c46e098caa35"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:55:38.437634 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.437623    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-token" pod "3fbe652f-e7ae-4938-bff9-c46e098caa35" (UID: "3fbe652f-e7ae-4938-bff9-c46e098caa35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:55:38.437634 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.437608    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-kube-api-access-v6ngm" pod "3fbe652f-e7ae-4938-bff9-c46e098caa35" (UID: "3fbe652f-e7ae-4938-bff9-c46e098caa35"). InnerVolumeSpecName "kube-api-access-v6ngm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:55:38.437791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.437701    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-dns-cert" pod "3fbe652f-e7ae-4938-bff9-c46e098caa35" (UID: "3fbe652f-e7ae-4938-bff9-c46e098caa35"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:55:38.437791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.437724    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbe652f-e7ae-4938-bff9-c46e098caa35-local-certs" pod "3fbe652f-e7ae-4938-bff9-c46e098caa35" (UID: "3fbe652f-e7ae-4938-bff9-c46e098caa35"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:55:38.537791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.537751    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:55:38.537791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.537785    5112 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-csr-dns-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:55:38.537791 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.537795    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ngm\" (UniqueName: \"kubernetes.io/projected/3fbe652f-e7ae-4938-bff9-c46e098caa35-kube-api-access-v6ngm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:55:38.538032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.537808    5112 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbe652f-e7ae-4938-bff9-c46e098caa35-local-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:55:38.538032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.537818    5112 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-istio-kubeconfig\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:55:38.538032 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.537827    5112 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3fbe652f-e7ae-4938-bff9-c46e098caa35-cacerts\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:55:38.626315 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.626278    5112 generic.go:356] "Generic (PLEG): container finished" podID="3fbe652f-e7ae-4938-bff9-c46e098caa35" containerID="55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4" exitCode=0
Nov 03 16:55:38.626518 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.626332    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"
Nov 03 16:55:38.626587 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.626520    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh" event={"ID":"3fbe652f-e7ae-4938-bff9-c46e098caa35","Type":"ContainerDied","Data":"55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4"}
Nov 03 16:55:38.626587 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.626557    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh" event={"ID":"3fbe652f-e7ae-4938-bff9-c46e098caa35","Type":"ContainerDied","Data":"3c45667da4e9267a1d643c91fd4610880140a32943f88f3f379dc3c10f5999fa"}
Nov 03 16:55:38.626711 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.626596    5112 scope.go:122] "RemoveContainer" containerID="55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4"
Nov 03 16:55:38.638638 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.638614    5112 scope.go:122] "RemoveContainer" containerID="55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4"
Nov 03 16:55:38.638934 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:55:38.638914    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4\": container with ID starting with 55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4 not found: ID does not exist" containerID="55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4"
Nov 03 16:55:38.639004 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.638942    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4"} err="failed to get container status \"55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4\": rpc error: code = NotFound desc = could not find container \"55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4\": container with ID starting with 55d2b96b5fe3eeb7e819e723dbff5cafadadee26d3b99ba7b20e782558c425f4 not found: ID does not exist"
Nov 03 16:55:38.789963 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.789872    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"]
Nov 03 16:55:38.796062 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:38.796032    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-km6lh"]
Nov 03 16:55:39.272311 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:39.272279    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:55:39.284486 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:39.284460    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:55:39.412883 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:39.412836    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 16:55:39.423812 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:39.423789    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 16:55:40.540338 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.540301    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"]
Nov 03 16:55:40.541083 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.541065    5112 cpu_manager.go:414] "containerMap: removing container" podUID="3fbe652f-e7ae-4938-bff9-c46e098caa35" containerName="discovery"
Nov 03 16:55:40.541083 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.541084    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbe652f-e7ae-4938-bff9-c46e098caa35" containerName="discovery"
Nov 03 16:55:40.541249 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.541230    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbe652f-e7ae-4938-bff9-c46e098caa35" containerName="discovery"
Nov 03 16:55:40.546314 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.546276    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.557259 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.557235    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbe652f-e7ae-4938-bff9-c46e098caa35" path="/var/lib/kubelet/pods/3fbe652f-e7ae-4938-bff9-c46e098caa35/volumes"
Nov 03 16:55:40.644755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.644722    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-61be78\"/\"openshift-service-ca.crt\""
Nov 03 16:55:40.644755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.644750    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-61be78\"/\"kube-root-ca.crt\""
Nov 03 16:55:40.645002 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.644835    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-61be78\"/\"test-model-61be78-kserve-self-signed-certs\""
Nov 03 16:55:40.645127 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.645094    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-61be78\"/\"default-dockercfg-r2gvb\""
Nov 03 16:55:40.647751 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.647726    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"]
Nov 03 16:55:40.673286 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.673255    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-dshm\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.673451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.673383    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfnb\" (UniqueName: \"kubernetes.io/projected/11a84535-dfc6-45fe-a0c1-97599e7a4adc-kube-api-access-brfnb\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.673451 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.673426    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-home\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.673585 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.673455    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tmp-dir\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.673703 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.673583    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-model-cache\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.673703 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.673631    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tls-certs\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.775365 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775329    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfnb\" (UniqueName: \"kubernetes.io/projected/11a84535-dfc6-45fe-a0c1-97599e7a4adc-kube-api-access-brfnb\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.775583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775393    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-home\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.775583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775430    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tmp-dir\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.775583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775524    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-model-cache\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.775583 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775570    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tls-certs\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.775820 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775690    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-dshm\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.775952 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775928    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-home\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.776041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.775980    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-model-cache\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.776041 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.776003    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tmp-dir\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.778036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.778009    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-dshm\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.778502 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.778482    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tls-certs\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.834241 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.834126    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfnb\" (UniqueName: \"kubernetes.io/projected/11a84535-dfc6-45fe-a0c1-97599e7a4adc-kube-api-access-brfnb\") pod \"test-model-61be78-kserve-dc46f4765-ppjxg\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") " pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:40.859980 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:40.859940    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:41.251283 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:41.251255    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"]
Nov 03 16:55:41.252297 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:55:41.252268    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11a84535_dfc6_45fe_a0c1_97599e7a4adc.slice/crio-3a351aef3ef41a86f03b5638c2c455c60b44303e652e4397b0a75fea63f09aa8 WatchSource:0}: Error finding container 3a351aef3ef41a86f03b5638c2c455c60b44303e652e4397b0a75fea63f09aa8: Status 404 returned error can't find the container with id 3a351aef3ef41a86f03b5638c2c455c60b44303e652e4397b0a75fea63f09aa8
Nov 03 16:55:41.654413 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:41.654373    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg" event={"ID":"11a84535-dfc6-45fe-a0c1-97599e7a4adc","Type":"ContainerStarted","Data":"6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62"}
Nov 03 16:55:41.654868 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:41.654427    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg" event={"ID":"11a84535-dfc6-45fe-a0c1-97599e7a4adc","Type":"ContainerStarted","Data":"3a351aef3ef41a86f03b5638c2c455c60b44303e652e4397b0a75fea63f09aa8"}
Nov 03 16:55:41.772646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:41.772579    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg" podStartSLOduration=1.772560524 podStartE2EDuration="1.772560524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:41.768072314 +0000 UTC m=+2703.347376169" watchObservedRunningTime="2026-11-03 16:55:41.772560524 +0000 UTC m=+2703.351864379"
Nov 03 16:55:41.860780 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:41.860743    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:46.218843 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.218805    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"]
Nov 03 16:55:46.223225 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.223190    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.279025 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.278992    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ddf492\"/\"kube-root-ca.crt\""
Nov 03 16:55:46.279025 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.279003    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ddf492\"/\"openshift-service-ca.crt\""
Nov 03 16:55:46.279264 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.279160    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ddf492\"/\"default-dockercfg-m7clg\""
Nov 03 16:55:46.326097 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.326063    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ddf492\"/\"test-model-ddf492-kserve-self-signed-certs\""
Nov 03 16:55:46.331178 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.331144    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"]
Nov 03 16:55:46.349064 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.349031    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-model-cache\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.349270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.349074    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-dshm\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.349270 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.349191    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tmp-dir\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.349401 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.349313    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tls-certs\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.349443 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.349405    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-kube-api-access-hl2pt\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.349443 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.349434    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-home\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.450607 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.450566    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-kube-api-access-hl2pt\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.450800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.450614    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-home\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.450800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.450702    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-model-cache\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.450800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.450728    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-dshm\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.450800 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.450774    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tmp-dir\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.451019 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.450821    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tls-certs\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.451148 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.451124    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-home\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.451247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.451134    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-model-cache\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.451247 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.451236    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tmp-dir\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.453036 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.453012    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-dshm\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.453378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.453360    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tls-certs\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.507964 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.507880    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-kube-api-access-hl2pt\") pod \"test-model-ddf492-kserve-6f6db4749-4nfcm\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") " pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.537428 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.537395    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:46.922589 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:46.922557    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"]
Nov 03 16:55:46.925231 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:55:46.925171    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1b2d25b_22c1_4a07_8d50_5a4c2c7294b6.slice/crio-1128aebc98871d20213bf8cdc7c069fdb085a1b8ac98f6d17fd059a00751bf8c WatchSource:0}: Error finding container 1128aebc98871d20213bf8cdc7c069fdb085a1b8ac98f6d17fd059a00751bf8c: Status 404 returned error can't find the container with id 1128aebc98871d20213bf8cdc7c069fdb085a1b8ac98f6d17fd059a00751bf8c
Nov 03 16:55:47.694063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:47.694023    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm" event={"ID":"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6","Type":"ContainerStarted","Data":"f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d"}
Nov 03 16:55:47.694543 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:47.694077    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm" event={"ID":"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6","Type":"ContainerStarted","Data":"1128aebc98871d20213bf8cdc7c069fdb085a1b8ac98f6d17fd059a00751bf8c"}
Nov 03 16:55:47.810397 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:47.810343    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm" podStartSLOduration=1.810329291 podStartE2EDuration="1.810329291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:47.808005664 +0000 UTC m=+2709.387309521" watchObservedRunningTime="2026-11-03 16:55:47.810329291 +0000 UTC m=+2709.389633183"
Nov 03 16:55:48.537527 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:48.537493    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:50.860758 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:50.860719    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:50.874073 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:50.874046    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:51.731699 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:51.731667    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:55:56.537674 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:56.537634    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:56.552518 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:56.552479    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:56.767681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:56.767648    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:55:57.668991 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.668958    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"]
Nov 03 16:55:57.673554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.673535    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.743494 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.743450    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"]
Nov 03 16:55:57.772640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.772609    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-963a2f\"/\"openshift-service-ca.crt\""
Nov 03 16:55:57.772640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.772631    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-963a2f\"/\"test-model-963a2f-kserve-self-signed-certs\""
Nov 03 16:55:57.772640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.772608    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-963a2f\"/\"default-dockercfg-9jq9v\""
Nov 03 16:55:57.772896 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.772647    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-963a2f\"/\"kube-root-ca.crt\""
Nov 03 16:55:57.793621 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.793588    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-dshm\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.793789 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.793743    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvtm8\" (UniqueName: \"kubernetes.io/projected/6ea32444-b868-4322-81b3-4fd19c09387b-kube-api-access-pvtm8\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.793867 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.793810    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-home\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.793930 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.793880    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea32444-b868-4322-81b3-4fd19c09387b-tls-certs\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.794162 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.793961    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-tmp-dir\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.794162 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.794009    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-model-cache\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895144 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895108    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-tmp-dir\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895355 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895163    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-model-cache\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895355 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895301    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-dshm\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895469 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895441    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvtm8\" (UniqueName: \"kubernetes.io/projected/6ea32444-b868-4322-81b3-4fd19c09387b-kube-api-access-pvtm8\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895534 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895509    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-home\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895621 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895605    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea32444-b868-4322-81b3-4fd19c09387b-tls-certs\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895681 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895645    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-model-cache\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895742 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895605    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-tmp-dir\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.895840 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.895820    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-home\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.897919 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.897891    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-dshm\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.898150 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.898126    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea32444-b868-4322-81b3-4fd19c09387b-tls-certs\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.952871 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.952810    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvtm8\" (UniqueName: \"kubernetes.io/projected/6ea32444-b868-4322-81b3-4fd19c09387b-kube-api-access-pvtm8\") pod \"test-model-963a2f-kserve-6454954665-gft4z\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") " pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:57.989066 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:57.989024    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:55:58.380883 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:58.380855    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"]
Nov 03 16:55:58.382538 ip-10-0-137-122 kubenswrapper[5112]: W1103 16:55:58.382503    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ea32444_b868_4322_81b3_4fd19c09387b.slice/crio-d721cad30145ba5e9196ae6afa09740e4473e47666c470d954bac5e5eaff9226 WatchSource:0}: Error finding container d721cad30145ba5e9196ae6afa09740e4473e47666c470d954bac5e5eaff9226: Status 404 returned error can't find the container with id d721cad30145ba5e9196ae6afa09740e4473e47666c470d954bac5e5eaff9226
Nov 03 16:55:58.771783 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:58.771741    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z" event={"ID":"6ea32444-b868-4322-81b3-4fd19c09387b","Type":"ContainerStarted","Data":"48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e"}
Nov 03 16:55:58.772235 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:58.771793    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z" event={"ID":"6ea32444-b868-4322-81b3-4fd19c09387b","Type":"ContainerStarted","Data":"d721cad30145ba5e9196ae6afa09740e4473e47666c470d954bac5e5eaff9226"}
Nov 03 16:55:58.889065 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:58.889005    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z" podStartSLOduration=1.8889913040000001 podStartE2EDuration="1.888991304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:58.886064473 +0000 UTC m=+2720.465368329" watchObservedRunningTime="2026-11-03 16:55:58.888991304 +0000 UTC m=+2720.468295156"
Nov 03 16:55:58.989476 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:55:58.989442    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:56:01.847946 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:01.847911    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"]
Nov 03 16:56:01.848448 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:01.848153    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" containerID="cri-o://88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a" gracePeriod=30
Nov 03 16:56:03.756970 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:03.756929    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-77r8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:56:03.757513 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:03.756995    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:56:07.128005 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.127942    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:56:07.202905 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.202868    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-envoy\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203107 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.202939    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-credential-socket\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203107 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203017    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istiod-ca-cert\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203107 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203088    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-token\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-token\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203328 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203113    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-socket\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203328 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203164    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-kube-api-access-wzsts\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-kube-api-access-wzsts\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203322    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-podinfo\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203432 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203354    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-certs\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.203554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203341    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-credential-socket" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:07.203554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203518    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istiod-ca-cert" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:56:07.203554 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203433    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-socket" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:07.203719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203626    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-certs" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:07.203719 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.203697    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-data\") pod \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\" (UID: \"2815b0ea-11fe-42fd-9b0f-e4f438ef3873\") "
Nov 03 16:56:07.204037 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.204010    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-data" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:07.204452 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.204429    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.204504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.204461    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.204504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.204476    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.204504 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.204491    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.204614 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.204504    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.205552 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.205521    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-podinfo" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:56:07.205647 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.205553    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-token" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:56:07.205647 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.205634    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-envoy" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:07.205730 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.205647    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-kube-api-access-wzsts" pod "2815b0ea-11fe-42fd-9b0f-e4f438ef3873" (UID: "2815b0ea-11fe-42fd-9b0f-e4f438ef3873"). InnerVolumeSpecName "kube-api-access-wzsts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:56:07.306044 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.306003    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.306044 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.306036    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzsts\" (UniqueName: \"kubernetes.io/projected/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-kube-api-access-wzsts\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.306044 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.306047    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.306308 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.306058    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2815b0ea-11fe-42fd-9b0f-e4f438ef3873-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:07.844190 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.844154    5112 generic.go:356] "Generic (PLEG): container finished" podID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerID="88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a" exitCode=0
Nov 03 16:56:07.844406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.844238    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" event={"ID":"2815b0ea-11fe-42fd-9b0f-e4f438ef3873","Type":"ContainerDied","Data":"88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a"}
Nov 03 16:56:07.844406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.844301    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r" event={"ID":"2815b0ea-11fe-42fd-9b0f-e4f438ef3873","Type":"ContainerDied","Data":"4c6b7d2d83d29ba7491b8e36d84139f94abbe7264cf9bb110d289e42c6a54d4b"}
Nov 03 16:56:07.844406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.844246    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"
Nov 03 16:56:07.844542 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.844327    5112 scope.go:122] "RemoveContainer" containerID="88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a"
Nov 03 16:56:07.856147 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.856125    5112 scope.go:122] "RemoveContainer" containerID="88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a"
Nov 03 16:56:07.856513 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:56:07.856480    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a\": container with ID starting with 88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a not found: ID does not exist" containerID="88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a"
Nov 03 16:56:07.856594 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.856523    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a"} err="failed to get container status \"88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a\": rpc error: code = NotFound desc = could not find container \"88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a\": container with ID starting with 88984dedf8dbdf8270973fef27bf1a772fd2b3b200f2b692076d7f16304e2a3a not found: ID does not exist"
Nov 03 16:56:07.884008 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:56:07.883969    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:56:07.989861 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:07.989824    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:56:08.003192 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:08.003160    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:56:08.056869 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:08.056834    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"]
Nov 03 16:56:08.104217 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:08.104111    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-77r8r"]
Nov 03 16:56:08.550779 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:08.550746    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" path="/var/lib/kubelet/pods/2815b0ea-11fe-42fd-9b0f-e4f438ef3873/volumes"
Nov 03 16:56:08.862549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:08.862472    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 16:56:37.885569 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:56:37.885528    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:56:47.256549 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:47.256512    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"]
Nov 03 16:56:47.257063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:47.256789    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" containerID="cri-o://a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7" gracePeriod=30
Nov 03 16:56:48.696932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:48.696887    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-t2mvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 16:56:48.697409 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:48.696953    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 16:56:52.573528 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.573489    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:56:52.681121 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681080    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-envoy\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681359 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681138    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-kube-api-access-259kx\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-kube-api-access-259kx\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681359 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681249    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-podinfo\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681359 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681290    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393e775e-2696-48b4-bc0b-5f5c34159a17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393e775e-2696-48b4-bc0b-5f5c34159a17-istiod-ca-cert\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681359 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681330    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-data\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681601 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681414    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-token\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-token\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681601 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681463    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-certs\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681601 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681496    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-socket\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681747 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681657    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-credential-socket\") pod \"393e775e-2696-48b4-bc0b-5f5c34159a17\" (UID: \"393e775e-2696-48b4-bc0b-5f5c34159a17\") "
Nov 03 16:56:52.681747 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.681688    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393e775e-2696-48b4-bc0b-5f5c34159a17-istiod-ca-cert" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:56:52.682118 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.682089    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-data" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.682266    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-credential-socket" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.682307    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-socket" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.682474    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-certs" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.682884    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.682995    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393e775e-2696-48b4-bc0b-5f5c34159a17-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.683017    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.683033    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.683085 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.683047    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.683867 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.683838    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-envoy" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:56:52.684074 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.684040    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-podinfo" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:56:52.684192 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.684122    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-kube-api-access-259kx" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "kube-api-access-259kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:56:52.684783 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.684760    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-token" pod "393e775e-2696-48b4-bc0b-5f5c34159a17" (UID: "393e775e-2696-48b4-bc0b-5f5c34159a17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:56:52.784418 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.784375    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.784418 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.784416    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.784688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.784432    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393e775e-2696-48b4-bc0b-5f5c34159a17-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:52.784688 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:52.784447    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-259kx\" (UniqueName: \"kubernetes.io/projected/393e775e-2696-48b4-bc0b-5f5c34159a17-kube-api-access-259kx\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:56:53.162378 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.162339    5112 generic.go:356] "Generic (PLEG): container finished" podID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerID="a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7" exitCode=0
Nov 03 16:56:53.162604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.162411    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"
Nov 03 16:56:53.162604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.162425    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" event={"ID":"393e775e-2696-48b4-bc0b-5f5c34159a17","Type":"ContainerDied","Data":"a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7"}
Nov 03 16:56:53.162604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.162521    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd" event={"ID":"393e775e-2696-48b4-bc0b-5f5c34159a17","Type":"ContainerDied","Data":"98432e56b02229eb5f8f90f062f5d4944b2d82a2af251e94243ea25969a9dd67"}
Nov 03 16:56:53.162604 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.162552    5112 scope.go:122] "RemoveContainer" containerID="a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7"
Nov 03 16:56:53.176087 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.176067    5112 scope.go:122] "RemoveContainer" containerID="a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7"
Nov 03 16:56:53.176445 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:56:53.176422    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7\": container with ID starting with a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7 not found: ID does not exist" containerID="a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7"
Nov 03 16:56:53.176515 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.176455    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7"} err="failed to get container status \"a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7\": rpc error: code = NotFound desc = could not find container \"a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7\": container with ID starting with a2204a781bafb69688cdb243c95288944fc0b04a3b6efe04c7be2c08cfa605d7 not found: ID does not exist"
Nov 03 16:56:53.325726 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.325685    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"]
Nov 03 16:56:53.372520 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:53.372487    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-t2mvd"]
Nov 03 16:56:54.550586 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:56:54.550556    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" path="/var/lib/kubelet/pods/393e775e-2696-48b4-bc0b-5f5c34159a17/volumes"
Nov 03 16:57:07.885514 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:57:07.885469    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:57:37.883771 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:57:37.883729    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:58:07.885575 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:58:07.885534    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:58:37.886340 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:58:37.886281    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:59:07.883364 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:59:07.883276    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:59:14.390278 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:14.390245    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"]
Nov 03 16:59:18.478406 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:18.478370    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"]
Nov 03 16:59:18.478862 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:18.478589    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy" containerID="cri-o://6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b" gracePeriod=30
Nov 03 16:59:23.745344 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.745270    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:59:23.780582 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780552    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-kube-api-access-97tmq\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-kube-api-access-97tmq\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.780708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780604    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-envoy\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.780708 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780646    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istiod-ca-cert\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.780808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780719    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-certs\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.780808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780744    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-socket\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.780808 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780774    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-data\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.781038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780830    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-podinfo\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.781038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780876    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-token\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-token\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.781038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.780906    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-credential-socket\") pod \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\" (UID: \"725cb813-4d0c-4922-b79c-ec7c2624d5a7\") "
Nov 03 16:59:23.781038 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.781023    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-socket" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:23.781304 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.781048    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-certs" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:23.781304 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.781137    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istiod-ca-cert" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:59:23.781948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.781665    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-credential-socket" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:23.782063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.781978    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-data" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:23.782063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.782050    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.782173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.782088    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.782173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.782106    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.782173 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.782122    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.785948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.784119    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-token" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:59:23.785948 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.784341    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-kube-api-access-97tmq" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "kube-api-access-97tmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:59:23.786141 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.786011    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-envoy" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:23.793379 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.793350    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-podinfo" pod "725cb813-4d0c-4922-b79c-ec7c2624d5a7" (UID: "725cb813-4d0c-4922-b79c-ec7c2624d5a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 16:59:23.883841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.883796    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.883841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.883827    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.883841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.883837    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.883841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.883846    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97tmq\" (UniqueName: \"kubernetes.io/projected/725cb813-4d0c-4922-b79c-ec7c2624d5a7-kube-api-access-97tmq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:23.883841 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:23.883855    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/725cb813-4d0c-4922-b79c-ec7c2624d5a7-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:24.195782 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.195741    5112 generic.go:356] "Generic (PLEG): container finished" podID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerID="6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b" exitCode=0
Nov 03 16:59:24.196012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.195816    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"
Nov 03 16:59:24.196012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.195830    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" event={"ID":"725cb813-4d0c-4922-b79c-ec7c2624d5a7","Type":"ContainerDied","Data":"6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b"}
Nov 03 16:59:24.196012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.195922    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7" event={"ID":"725cb813-4d0c-4922-b79c-ec7c2624d5a7","Type":"ContainerDied","Data":"c9fe46ec37c3cb135430d4016f4772ac2d9c815ff1f3b9f69862c89df55928ca"}
Nov 03 16:59:24.196012 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.195945    5112 scope.go:122] "RemoveContainer" containerID="6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b"
Nov 03 16:59:24.206779 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.206755    5112 scope.go:122] "RemoveContainer" containerID="50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51"
Nov 03 16:59:24.216646 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.216628    5112 scope.go:122] "RemoveContainer" containerID="6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b"
Nov 03 16:59:24.216884 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:59:24.216867    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b\": container with ID starting with 6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b not found: ID does not exist" containerID="6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b"
Nov 03 16:59:24.216947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.216890    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b"} err="failed to get container status \"6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b\": rpc error: code = NotFound desc = could not find container \"6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b\": container with ID starting with 6adfc649156a4a9194ba1fb23ee76bdfa7e4db8bb1ed184e2b63dbf26c6cec1b not found: ID does not exist"
Nov 03 16:59:24.216947 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.216907    5112 scope.go:122] "RemoveContainer" containerID="50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51"
Nov 03 16:59:24.217110 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:59:24.217095    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51\": container with ID starting with 50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51 not found: ID does not exist" containerID="50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51"
Nov 03 16:59:24.217171 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.217113    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51"} err="failed to get container status \"50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51\": rpc error: code = NotFound desc = could not find container \"50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51\": container with ID starting with 50e1816f50183e4bc304522b9bf205d99296f6f2b9c046656715b8506bfa3c51 not found: ID does not exist"
Nov 03 16:59:24.358185 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.358152    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"]
Nov 03 16:59:24.364010 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.363983    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sbbf7"]
Nov 03 16:59:24.549772 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:24.549697    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" path="/var/lib/kubelet/pods/725cb813-4d0c-4922-b79c-ec7c2624d5a7/volumes"
Nov 03 16:59:29.410832 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.410785    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg" podUID="11a84535-dfc6-45fe-a0c1-97599e7a4adc" containerName="main" containerID="cri-o://6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62" gracePeriod=45
Nov 03 16:59:29.691673 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.691642    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:59:29.755458 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755428    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-model-cache\") pod \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") "
Nov 03 16:59:29.755670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755485    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-home\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-home\") pod \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") "
Nov 03 16:59:29.755670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755560    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a84535-dfc6-45fe-a0c1-97599e7a4adc-kube-api-access-brfnb\" (UniqueName: \"kubernetes.io/projected/11a84535-dfc6-45fe-a0c1-97599e7a4adc-kube-api-access-brfnb\") pod \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") "
Nov 03 16:59:29.755670 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755629    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tmp-dir\") pod \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") "
Nov 03 16:59:29.755847 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755679    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-dshm\") pod \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") "
Nov 03 16:59:29.755847 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755788    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-model-cache" pod "11a84535-dfc6-45fe-a0c1-97599e7a4adc" (UID: "11a84535-dfc6-45fe-a0c1-97599e7a4adc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:29.755847 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755800    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-home" pod "11a84535-dfc6-45fe-a0c1-97599e7a4adc" (UID: "11a84535-dfc6-45fe-a0c1-97599e7a4adc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:29.755847 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755811    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tls-certs\" (UniqueName: \"kubernetes.io/secret/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tls-certs\") pod \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\" (UID: \"11a84535-dfc6-45fe-a0c1-97599e7a4adc\") "
Nov 03 16:59:29.755997 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.755879    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tmp-dir" pod "11a84535-dfc6-45fe-a0c1-97599e7a4adc" (UID: "11a84535-dfc6-45fe-a0c1-97599e7a4adc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:29.756325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.756300    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:29.756325 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.756326    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:29.756540 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.756337    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:29.758131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.758106    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-dshm" pod "11a84535-dfc6-45fe-a0c1-97599e7a4adc" (UID: "11a84535-dfc6-45fe-a0c1-97599e7a4adc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:29.758131 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.758118    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tls-certs" pod "11a84535-dfc6-45fe-a0c1-97599e7a4adc" (UID: "11a84535-dfc6-45fe-a0c1-97599e7a4adc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:59:29.758350 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.758144    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a84535-dfc6-45fe-a0c1-97599e7a4adc-kube-api-access-brfnb" pod "11a84535-dfc6-45fe-a0c1-97599e7a4adc" (UID: "11a84535-dfc6-45fe-a0c1-97599e7a4adc"). InnerVolumeSpecName "kube-api-access-brfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:59:29.856984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.856950    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11a84535-dfc6-45fe-a0c1-97599e7a4adc-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:29.856984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.856979    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfnb\" (UniqueName: \"kubernetes.io/projected/11a84535-dfc6-45fe-a0c1-97599e7a4adc-kube-api-access-brfnb\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:29.856984 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:29.856990    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11a84535-dfc6-45fe-a0c1-97599e7a4adc-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:30.237973 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.237941    5112 generic.go:356] "Generic (PLEG): container finished" podID="11a84535-dfc6-45fe-a0c1-97599e7a4adc" containerID="6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62" exitCode=0
Nov 03 16:59:30.238172 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.237993    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"
Nov 03 16:59:30.238172 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.238025    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg" event={"ID":"11a84535-dfc6-45fe-a0c1-97599e7a4adc","Type":"ContainerDied","Data":"6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62"}
Nov 03 16:59:30.238172 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.238068    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg" event={"ID":"11a84535-dfc6-45fe-a0c1-97599e7a4adc","Type":"ContainerDied","Data":"3a351aef3ef41a86f03b5638c2c455c60b44303e652e4397b0a75fea63f09aa8"}
Nov 03 16:59:30.238172 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.238087    5112 scope.go:122] "RemoveContainer" containerID="6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62"
Nov 03 16:59:30.248916 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.248896    5112 scope.go:122] "RemoveContainer" containerID="6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62"
Nov 03 16:59:30.249253 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:59:30.249227    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62\": container with ID starting with 6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62 not found: ID does not exist" containerID="6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62"
Nov 03 16:59:30.249348 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.249259    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62"} err="failed to get container status \"6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62\": rpc error: code = NotFound desc = could not find container \"6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62\": container with ID starting with 6ae65cf07cc91905e4c64658e011cd08da95158ad1381453623728e995235f62 not found: ID does not exist"
Nov 03 16:59:30.400629 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.400587    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"]
Nov 03 16:59:30.404804 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.404771    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-61be78/test-model-61be78-kserve-dc46f4765-ppjxg"]
Nov 03 16:59:30.550416 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:30.550324    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a84535-dfc6-45fe-a0c1-97599e7a4adc" path="/var/lib/kubelet/pods/11a84535-dfc6-45fe-a0c1-97599e7a4adc/volumes"
Nov 03 16:59:35.624430 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:35.624394    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"]
Nov 03 16:59:37.450755 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.450718    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"]
Nov 03 16:59:37.451189 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.450924    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj" podUID="2144027f-280e-486c-a857-1efab523a47d" containerName="maas-api" containerID="cri-o://224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254" gracePeriod=30
Nov 03 16:59:37.729474 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.729446    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:59:37.857473 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.857436    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2144027f-280e-486c-a857-1efab523a47d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2144027f-280e-486c-a857-1efab523a47d-openshift-service-ca\") pod \"2144027f-280e-486c-a857-1efab523a47d\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") "
Nov 03 16:59:37.857724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.857574    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-metrics-certs\") pod \"2144027f-280e-486c-a857-1efab523a47d\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") "
Nov 03 16:59:37.857724 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.857628    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-maas-api-tls\") pod \"2144027f-280e-486c-a857-1efab523a47d\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") "
Nov 03 16:59:37.857870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.857796    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2144027f-280e-486c-a857-1efab523a47d-kube-api-access-kpvmf\" (UniqueName: \"kubernetes.io/projected/2144027f-280e-486c-a857-1efab523a47d-kube-api-access-kpvmf\") pod \"2144027f-280e-486c-a857-1efab523a47d\" (UID: \"2144027f-280e-486c-a857-1efab523a47d\") "
Nov 03 16:59:37.857870 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.857846    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2144027f-280e-486c-a857-1efab523a47d-openshift-service-ca" pod "2144027f-280e-486c-a857-1efab523a47d" (UID: "2144027f-280e-486c-a857-1efab523a47d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 16:59:37.858245 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.858175    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2144027f-280e-486c-a857-1efab523a47d-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:37.860521 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.860486    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-metrics-certs" pod "2144027f-280e-486c-a857-1efab523a47d" (UID: "2144027f-280e-486c-a857-1efab523a47d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:59:37.860640 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.860548    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-maas-api-tls" pod "2144027f-280e-486c-a857-1efab523a47d" (UID: "2144027f-280e-486c-a857-1efab523a47d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:59:37.860715 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.860663    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2144027f-280e-486c-a857-1efab523a47d-kube-api-access-kpvmf" pod "2144027f-280e-486c-a857-1efab523a47d" (UID: "2144027f-280e-486c-a857-1efab523a47d"). InnerVolumeSpecName "kube-api-access-kpvmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:59:37.890687 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:59:37.890611    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 16:59:37.959243 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.959105    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:37.959243 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.959139    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvmf\" (UniqueName: \"kubernetes.io/projected/2144027f-280e-486c-a857-1efab523a47d-kube-api-access-kpvmf\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:37.959243 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:37.959154    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2144027f-280e-486c-a857-1efab523a47d-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:38.291258 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.291150    5112 generic.go:356] "Generic (PLEG): container finished" podID="2144027f-280e-486c-a857-1efab523a47d" containerID="224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254" exitCode=0
Nov 03 16:59:38.291258 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.291235    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"
Nov 03 16:59:38.291258 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.291241    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj" event={"ID":"2144027f-280e-486c-a857-1efab523a47d","Type":"ContainerDied","Data":"224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254"}
Nov 03 16:59:38.291484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.291307    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj" event={"ID":"2144027f-280e-486c-a857-1efab523a47d","Type":"ContainerDied","Data":"413c545effc3b9cb7cfe22634470af9cacb41d4d145516d8f210cedea065c1ef"}
Nov 03 16:59:38.291484 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.291331    5112 scope.go:122] "RemoveContainer" containerID="224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254"
Nov 03 16:59:38.302585 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.302566    5112 scope.go:122] "RemoveContainer" containerID="224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254"
Nov 03 16:59:38.302867 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:59:38.302846    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254\": container with ID starting with 224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254 not found: ID does not exist" containerID="224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254"
Nov 03 16:59:38.302932 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.302875    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254"} err="failed to get container status \"224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254\": rpc error: code = NotFound desc = could not find container \"224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254\": container with ID starting with 224174391d3a4efad6742e19838b7b7dea38461c55dd8e2458c9eba81fe6c254 not found: ID does not exist"
Nov 03 16:59:38.454040 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.454006    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"]
Nov 03 16:59:38.501105 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.501070    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-244c65-89bc69d69-gwdcj"]
Nov 03 16:59:38.550718 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:38.550643    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2144027f-280e-486c-a857-1efab523a47d" path="/var/lib/kubelet/pods/2144027f-280e-486c-a857-1efab523a47d/volumes"
Nov 03 16:59:50.644888 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:50.644847    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm" podUID="e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" containerName="main" containerID="cri-o://f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d" gracePeriod=45
Nov 03 16:59:50.756819 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:50.756776    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm" podUID="e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.140:8000/ready\": dial tcp 10.132.0.140:8000: connect: connection refused"
Nov 03 16:59:50.925980 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:50.925953    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:59:51.009334 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009292    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tls-certs\") pod \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") "
Nov 03 16:59:51.009539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009416    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tmp-dir\") pod \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") "
Nov 03 16:59:51.009539 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009497    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-kube-api-access-hl2pt\") pod \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") "
Nov 03 16:59:51.009663 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009557    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-dshm\") pod \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") "
Nov 03 16:59:51.009663 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009583    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-model-cache\") pod \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") "
Nov 03 16:59:51.009663 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009631    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-home\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-home\") pod \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\" (UID: \"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6\") "
Nov 03 16:59:51.009814 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009724    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tmp-dir" pod "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" (UID: "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:51.009916 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009885    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-model-cache" pod "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" (UID: "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:51.010020 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.009995    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-home" pod "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" (UID: "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:51.010170 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.010150    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:51.010285 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.010177    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:51.010285 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.010192    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:51.011817 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.011789    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tls-certs" pod "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" (UID: "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 16:59:51.012063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.011973    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-dshm" pod "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" (UID: "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 16:59:51.012063 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.012037    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-kube-api-access-hl2pt" pod "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" (UID: "e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6"). InnerVolumeSpecName "kube-api-access-hl2pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 16:59:51.110884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.110844    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl2pt\" (UniqueName: \"kubernetes.io/projected/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-kube-api-access-hl2pt\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:51.110884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.110875    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:51.110884 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.110886    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 16:59:51.401497 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.401459    5112 generic.go:356] "Generic (PLEG): container finished" podID="e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" containerID="f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d" exitCode=0
Nov 03 16:59:51.401679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.401510    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"
Nov 03 16:59:51.401679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.401532    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm" event={"ID":"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6","Type":"ContainerDied","Data":"f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d"}
Nov 03 16:59:51.401679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.401581    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm" event={"ID":"e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6","Type":"ContainerDied","Data":"1128aebc98871d20213bf8cdc7c069fdb085a1b8ac98f6d17fd059a00751bf8c"}
Nov 03 16:59:51.401679 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.401605    5112 scope.go:122] "RemoveContainer" containerID="f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d"
Nov 03 16:59:51.415693 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.415665    5112 scope.go:122] "RemoveContainer" containerID="f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d"
Nov 03 16:59:51.416145 ip-10-0-137-122 kubenswrapper[5112]: E1103 16:59:51.416121    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d\": container with ID starting with f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d not found: ID does not exist" containerID="f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d"
Nov 03 16:59:51.416260 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.416153    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d"} err="failed to get container status \"f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d\": rpc error: code = NotFound desc = could not find container \"f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d\": container with ID starting with f157b9e284ea7cde6fcc097073477f17ed11525e4b4c53d4af534857b6ee838d not found: ID does not exist"
Nov 03 16:59:51.565515 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.565476    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"]
Nov 03 16:59:51.571431 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:51.571386    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ddf492/test-model-ddf492-kserve-6f6db4749-4nfcm"]
Nov 03 16:59:52.549823 ip-10-0-137-122 kubenswrapper[5112]: I1103 16:59:52.549779    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" path="/var/lib/kubelet/pods/e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6/volumes"
Nov 03 17:00:00.217927 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.217882    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"]
Nov 03 17:00:00.218793 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218769    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2144027f-280e-486c-a857-1efab523a47d" containerName="maas-api"
Nov 03 17:00:00.218842 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218799    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2144027f-280e-486c-a857-1efab523a47d" containerName="maas-api"
Nov 03 17:00:00.218842 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218813    5112 cpu_manager.go:414] "containerMap: removing container" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy"
Nov 03 17:00:00.218842 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218821    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy"
Nov 03 17:00:00.218842 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218839    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy"
Nov 03 17:00:00.218990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218848    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy"
Nov 03 17:00:00.218990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218896    5112 cpu_manager.go:414] "containerMap: removing container" podUID="11a84535-dfc6-45fe-a0c1-97599e7a4adc" containerName="main"
Nov 03 17:00:00.218990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218904    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a84535-dfc6-45fe-a0c1-97599e7a4adc" containerName="main"
Nov 03 17:00:00.218990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218927    5112 cpu_manager.go:414] "containerMap: removing container" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy"
Nov 03 17:00:00.218990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218935    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy"
Nov 03 17:00:00.218990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218950    5112 cpu_manager.go:414] "containerMap: removing container" podUID="e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" containerName="main"
Nov 03 17:00:00.218990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.218957    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" containerName="main"
Nov 03 17:00:00.219274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.219103    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="393e775e-2696-48b4-bc0b-5f5c34159a17" containerName="istio-proxy"
Nov 03 17:00:00.219274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.219120    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b2d25b-22c1-4a07-8d50-5a4c2c7294b6" containerName="main"
Nov 03 17:00:00.219274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.219134    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2144027f-280e-486c-a857-1efab523a47d" containerName="maas-api"
Nov 03 17:00:00.219274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.219144    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy"
Nov 03 17:00:00.219274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.219153    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2815b0ea-11fe-42fd-9b0f-e4f438ef3873" containerName="istio-proxy"
Nov 03 17:00:00.219274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.219171    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a84535-dfc6-45fe-a0c1-97599e7a4adc" containerName="main"
Nov 03 17:00:00.224675 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.224654    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.237789 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.237763    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"]
Nov 03 17:00:00.238593 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.238577    5112 cpu_manager.go:414] "containerMap: removing container" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy"
Nov 03 17:00:00.238658 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.238595    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy"
Nov 03 17:00:00.238746 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.238735    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="725cb813-4d0c-4922-b79c-ec7c2624d5a7" containerName="istio-proxy"
Nov 03 17:00:00.242638 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.242616    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.286852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.286821    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:00:00.294382 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.294347    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"]
Nov 03 17:00:00.314817 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.314785    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gq7\" (UniqueName: \"kubernetes.io/projected/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-kube-api-access-f8gq7\") pod \"maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.314995 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.314914    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67ba266-b9d5-4bf6-9444-a858edd52bb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.315048 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.315009    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2bj\" (UniqueName: \"kubernetes.io/projected/b67ba266-b9d5-4bf6-9444-a858edd52bb0-kube-api-access-lg2bj\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.315085 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.315047    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.325561 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.325529    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"]
Nov 03 17:00:00.416736 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.416704    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2bj\" (UniqueName: \"kubernetes.io/projected/b67ba266-b9d5-4bf6-9444-a858edd52bb0-kube-api-access-lg2bj\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.416919 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.416751    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.416919 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.416880    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gq7\" (UniqueName: \"kubernetes.io/projected/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-kube-api-access-f8gq7\") pod \"maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.417048 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.416970    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67ba266-b9d5-4bf6-9444-a858edd52bb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.417551 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.417527    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.417663 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.417621    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67ba266-b9d5-4bf6-9444-a858edd52bb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.474194 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.474119    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gq7\" (UniqueName: \"kubernetes.io/projected/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-kube-api-access-f8gq7\") pod \"maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.474342 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.474307    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2bj\" (UniqueName: \"kubernetes.io/projected/b67ba266-b9d5-4bf6-9444-a858edd52bb0-kube-api-access-lg2bj\") pod \"maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.540379 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.540342    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:00.563697 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.563660    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:00.916847 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.916810    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"]
Nov 03 17:00:00.918277 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:00:00.918249    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2917f91_5099_4d4b_9e7e_2d7c95ebb38f.slice/crio-d58c9b8ad8e76a9c6540cee23c96099c92709a48443401b60442c878d2b89e9c WatchSource:0}: Error finding container d58c9b8ad8e76a9c6540cee23c96099c92709a48443401b60442c878d2b89e9c: Status 404 returned error can't find the container with id d58c9b8ad8e76a9c6540cee23c96099c92709a48443401b60442c878d2b89e9c
Nov 03 17:00:00.964274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:00.964248    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"]
Nov 03 17:00:00.965370 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:00:00.965343    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67ba266_b9d5_4bf6_9444_a858edd52bb0.slice/crio-d198ad624f5ecbe119c3feef33c469725eca9a9f5643d242d63fadcce589898c WatchSource:0}: Error finding container d198ad624f5ecbe119c3feef33c469725eca9a9f5643d242d63fadcce589898c: Status 404 returned error can't find the container with id d198ad624f5ecbe119c3feef33c469725eca9a9f5643d242d63fadcce589898c
Nov 03 17:00:01.472011 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:01.471902    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" event={"ID":"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f","Type":"ContainerStarted","Data":"83378afdabd7a15abf409905bb68b6cedaa19035f31af0b3065b7b8f1260960a"}
Nov 03 17:00:01.472011 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:01.471974    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" event={"ID":"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f","Type":"ContainerStarted","Data":"d58c9b8ad8e76a9c6540cee23c96099c92709a48443401b60442c878d2b89e9c"}
Nov 03 17:00:01.473550 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:01.473521    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" event={"ID":"b67ba266-b9d5-4bf6-9444-a858edd52bb0","Type":"ContainerStarted","Data":"2313d0700f547d37418304d6b66b9cf2cd0534a484aa88030da3212e8ca9c7fe"}
Nov 03 17:00:01.473687 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:01.473571    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" event={"ID":"b67ba266-b9d5-4bf6-9444-a858edd52bb0","Type":"ContainerStarted","Data":"d198ad624f5ecbe119c3feef33c469725eca9a9f5643d242d63fadcce589898c"}
Nov 03 17:00:01.584233 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:01.584163    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" podStartSLOduration=1.584145782 podStartE2EDuration="1.584145782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:01.583460635 +0000 UTC m=+2963.162764491" watchObservedRunningTime="2026-11-03 17:00:01.584145782 +0000 UTC m=+2963.163449641"
Nov 03 17:00:01.696720 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:01.696663    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" podStartSLOduration=1.696650425 podStartE2EDuration="1.696650425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:01.695856359 +0000 UTC m=+2963.275160232" watchObservedRunningTime="2026-11-03 17:00:01.696650425 +0000 UTC m=+2963.275954279"
Nov 03 17:00:02.497277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:02.494810    5112 generic.go:356] "Generic (PLEG): container finished" podID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerID="83378afdabd7a15abf409905bb68b6cedaa19035f31af0b3065b7b8f1260960a" exitCode=60
Nov 03 17:00:02.497277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:02.494995    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" event={"ID":"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f","Type":"ContainerDied","Data":"83378afdabd7a15abf409905bb68b6cedaa19035f31af0b3065b7b8f1260960a"}
Nov 03 17:00:02.497277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:02.495536    5112 scope.go:122] "RemoveContainer" containerID="83378afdabd7a15abf409905bb68b6cedaa19035f31af0b3065b7b8f1260960a"
Nov 03 17:00:02.507555 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:02.507521    5112 generic.go:356] "Generic (PLEG): container finished" podID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerID="2313d0700f547d37418304d6b66b9cf2cd0534a484aa88030da3212e8ca9c7fe" exitCode=60
Nov 03 17:00:02.507689 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:02.507594    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" event={"ID":"b67ba266-b9d5-4bf6-9444-a858edd52bb0","Type":"ContainerDied","Data":"2313d0700f547d37418304d6b66b9cf2cd0534a484aa88030da3212e8ca9c7fe"}
Nov 03 17:00:02.507922 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:02.507898    5112 scope.go:122] "RemoveContainer" containerID="2313d0700f547d37418304d6b66b9cf2cd0534a484aa88030da3212e8ca9c7fe"
Nov 03 17:00:03.519670 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:03.519634    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" event={"ID":"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f","Type":"ContainerStarted","Data":"78c69fc12cd3b0f90ea85d1e90b13da212ae14b59740196d609da9fc8c3c7dc7"}
Nov 03 17:00:03.521838 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:03.521801    5112 generic.go:356] "Generic (PLEG): container finished" podID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerID="2d3eca69adeb15748d94d085a9f64ef18d2d0c58688b8e1d29df913c92f9e477" exitCode=60
Nov 03 17:00:03.521987 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:03.521865    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" event={"ID":"b67ba266-b9d5-4bf6-9444-a858edd52bb0","Type":"ContainerDied","Data":"2d3eca69adeb15748d94d085a9f64ef18d2d0c58688b8e1d29df913c92f9e477"}
Nov 03 17:00:03.521987 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:03.521908    5112 scope.go:122] "RemoveContainer" containerID="2313d0700f547d37418304d6b66b9cf2cd0534a484aa88030da3212e8ca9c7fe"
Nov 03 17:00:03.522182 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:03.522162    5112 scope.go:122] "RemoveContainer" containerID="2d3eca69adeb15748d94d085a9f64ef18d2d0c58688b8e1d29df913c92f9e477"
Nov 03 17:00:03.522563 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:03.522539    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99_odh-ai-gateway-infra(b67ba266-b9d5-4bf6-9444-a858edd52bb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0"
Nov 03 17:00:04.531268 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:04.531236    5112 scope.go:122] "RemoveContainer" containerID="2d3eca69adeb15748d94d085a9f64ef18d2d0c58688b8e1d29df913c92f9e477"
Nov 03 17:00:04.531730 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:04.531552    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99_odh-ai-gateway-infra(b67ba266-b9d5-4bf6-9444-a858edd52bb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0"
Nov 03 17:00:04.533023 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:04.532999    5112 generic.go:356] "Generic (PLEG): container finished" podID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerID="78c69fc12cd3b0f90ea85d1e90b13da212ae14b59740196d609da9fc8c3c7dc7" exitCode=60
Nov 03 17:00:04.533132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:04.533058    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" event={"ID":"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f","Type":"ContainerDied","Data":"78c69fc12cd3b0f90ea85d1e90b13da212ae14b59740196d609da9fc8c3c7dc7"}
Nov 03 17:00:04.533132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:04.533123    5112 scope.go:122] "RemoveContainer" containerID="83378afdabd7a15abf409905bb68b6cedaa19035f31af0b3065b7b8f1260960a"
Nov 03 17:00:04.533324 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:04.533309    5112 scope.go:122] "RemoveContainer" containerID="78c69fc12cd3b0f90ea85d1e90b13da212ae14b59740196d609da9fc8c3c7dc7"
Nov 03 17:00:04.533616 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:04.533592    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l_odh-ai-gateway-infra(b2917f91-5099-4d4b-9e7e-2d7c95ebb38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f"
Nov 03 17:00:05.542437 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:05.542399    5112 scope.go:122] "RemoveContainer" containerID="78c69fc12cd3b0f90ea85d1e90b13da212ae14b59740196d609da9fc8c3c7dc7"
Nov 03 17:00:05.542821 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:05.542625    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l_odh-ai-gateway-infra(b2917f91-5099-4d4b-9e7e-2d7c95ebb38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f"
Nov 03 17:00:06.753681 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:06.753643    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"]
Nov 03 17:00:07.884650 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:07.884604    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:00:13.542271 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:13.542236    5112 scope.go:122] "RemoveContainer" containerID="2d3eca69adeb15748d94d085a9f64ef18d2d0c58688b8e1d29df913c92f9e477"
Nov 03 17:00:14.605265 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:14.605228    5112 generic.go:356] "Generic (PLEG): container finished" podID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerID="c735f6df3013d67f132b8462afb11139ece00f8bd375a18135dfad8321373a8d" exitCode=60
Nov 03 17:00:14.605739 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:14.605294    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" event={"ID":"b67ba266-b9d5-4bf6-9444-a858edd52bb0","Type":"ContainerDied","Data":"c735f6df3013d67f132b8462afb11139ece00f8bd375a18135dfad8321373a8d"}
Nov 03 17:00:14.605739 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:14.605339    5112 scope.go:122] "RemoveContainer" containerID="2d3eca69adeb15748d94d085a9f64ef18d2d0c58688b8e1d29df913c92f9e477"
Nov 03 17:00:14.605739 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:14.605711    5112 scope.go:122] "RemoveContainer" containerID="c735f6df3013d67f132b8462afb11139ece00f8bd375a18135dfad8321373a8d"
Nov 03 17:00:14.606011 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:14.605986    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99_odh-ai-gateway-infra(b67ba266-b9d5-4bf6-9444-a858edd52bb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0"
Nov 03 17:00:15.733594 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:15.733560    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"]
Nov 03 17:00:15.899553 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:15.899521    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:15.908504 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:15.908482    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67ba266-b9d5-4bf6-9444-a858edd52bb0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67ba266-b9d5-4bf6-9444-a858edd52bb0-maas-api-service-ca\") pod \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") "
Nov 03 17:00:15.908637 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:15.908575    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67ba266-b9d5-4bf6-9444-a858edd52bb0-kube-api-access-lg2bj\" (UniqueName: \"kubernetes.io/projected/b67ba266-b9d5-4bf6-9444-a858edd52bb0-kube-api-access-lg2bj\") pod \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\" (UID: \"b67ba266-b9d5-4bf6-9444-a858edd52bb0\") "
Nov 03 17:00:15.908854 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:15.908829    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67ba266-b9d5-4bf6-9444-a858edd52bb0-maas-api-service-ca" pod "b67ba266-b9d5-4bf6-9444-a858edd52bb0" (UID: "b67ba266-b9d5-4bf6-9444-a858edd52bb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:00:15.910693 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:15.910672    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67ba266-b9d5-4bf6-9444-a858edd52bb0-kube-api-access-lg2bj" pod "b67ba266-b9d5-4bf6-9444-a858edd52bb0" (UID: "b67ba266-b9d5-4bf6-9444-a858edd52bb0"). InnerVolumeSpecName "kube-api-access-lg2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:00:16.009690 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:16.009632    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b67ba266-b9d5-4bf6-9444-a858edd52bb0-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:16.009690 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:16.009656    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg2bj\" (UniqueName: \"kubernetes.io/projected/b67ba266-b9d5-4bf6-9444-a858edd52bb0-kube-api-access-lg2bj\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:16.620502 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:16.620472    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"
Nov 03 17:00:16.620663 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:16.620517    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99" event={"ID":"b67ba266-b9d5-4bf6-9444-a858edd52bb0","Type":"ContainerDied","Data":"d198ad624f5ecbe119c3feef33c469725eca9a9f5643d242d63fadcce589898c"}
Nov 03 17:00:16.620663 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:16.620581    5112 scope.go:122] "RemoveContainer" containerID="c735f6df3013d67f132b8462afb11139ece00f8bd375a18135dfad8321373a8d"
Nov 03 17:00:16.795780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:16.795742    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"]
Nov 03 17:00:16.842736 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:16.842657    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ddf492-29895420-b9f99"]
Nov 03 17:00:17.542814 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:17.542775    5112 scope.go:122] "RemoveContainer" containerID="78c69fc12cd3b0f90ea85d1e90b13da212ae14b59740196d609da9fc8c3c7dc7"
Nov 03 17:00:18.551384 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:18.551340    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" path="/var/lib/kubelet/pods/b67ba266-b9d5-4bf6-9444-a858edd52bb0/volumes"
Nov 03 17:00:18.635721 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:18.635631    5112 generic.go:356] "Generic (PLEG): container finished" podID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerID="1a774c472456dc53a11bc98efd688384aa86850f4ad3c5d53e906d381eec55ed" exitCode=60
Nov 03 17:00:18.635721 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:18.635701    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" event={"ID":"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f","Type":"ContainerDied","Data":"1a774c472456dc53a11bc98efd688384aa86850f4ad3c5d53e906d381eec55ed"}
Nov 03 17:00:18.635952 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:18.635774    5112 scope.go:122] "RemoveContainer" containerID="78c69fc12cd3b0f90ea85d1e90b13da212ae14b59740196d609da9fc8c3c7dc7"
Nov 03 17:00:18.636232 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:18.636187    5112 scope.go:122] "RemoveContainer" containerID="1a774c472456dc53a11bc98efd688384aa86850f4ad3c5d53e906d381eec55ed"
Nov 03 17:00:18.636517 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:18.636498    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l_odh-ai-gateway-infra(b2917f91-5099-4d4b-9e7e-2d7c95ebb38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f"
Nov 03 17:00:19.765929 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:19.765885    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"]
Nov 03 17:00:19.932175 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:19.932146    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:19.953771 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:19.953743    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-kube-api-access-f8gq7\" (UniqueName: \"kubernetes.io/projected/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-kube-api-access-f8gq7\") pod \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") "
Nov 03 17:00:19.953931 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:19.953781    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-maas-api-service-ca\") pod \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\" (UID: \"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f\") "
Nov 03 17:00:19.954261 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:19.954225    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-maas-api-service-ca" pod "b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" (UID: "b2917f91-5099-4d4b-9e7e-2d7c95ebb38f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:00:19.954403 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:19.954385    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:19.956483 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:19.956456    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-kube-api-access-f8gq7" pod "b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" (UID: "b2917f91-5099-4d4b-9e7e-2d7c95ebb38f"). InnerVolumeSpecName "kube-api-access-f8gq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:00:20.055260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:20.055157    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8gq7\" (UniqueName: \"kubernetes.io/projected/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f-kube-api-access-f8gq7\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:20.652780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:20.652748    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l" event={"ID":"b2917f91-5099-4d4b-9e7e-2d7c95ebb38f","Type":"ContainerDied","Data":"d58c9b8ad8e76a9c6540cee23c96099c92709a48443401b60442c878d2b89e9c"}
Nov 03 17:00:20.652976 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:20.652796    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"
Nov 03 17:00:20.652976 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:20.652808    5112 scope.go:122] "RemoveContainer" containerID="1a774c472456dc53a11bc98efd688384aa86850f4ad3c5d53e906d381eec55ed"
Nov 03 17:00:20.814261 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:20.814176    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"]
Nov 03 17:00:20.820544 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:20.820512    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-963a2f-29895420-6v72l"]
Nov 03 17:00:21.774920 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:21.774854    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z" podUID="6ea32444-b868-4322-81b3-4fd19c09387b" containerName="main" containerID="cri-o://48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e" gracePeriod=45
Nov 03 17:00:21.852909 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:21.852870    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z" podUID="6ea32444-b868-4322-81b3-4fd19c09387b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.141:8000/ready\": dial tcp 10.132.0.141:8000: connect: connection refused"
Nov 03 17:00:22.065331 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.065293    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 17:00:22.181466 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.181419    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea32444-b868-4322-81b3-4fd19c09387b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea32444-b868-4322-81b3-4fd19c09387b-tls-certs\") pod \"6ea32444-b868-4322-81b3-4fd19c09387b\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") "
Nov 03 17:00:22.181667 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.181546    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-tmp-dir\") pod \"6ea32444-b868-4322-81b3-4fd19c09387b\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") "
Nov 03 17:00:22.181667 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.181619    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea32444-b868-4322-81b3-4fd19c09387b-kube-api-access-pvtm8\" (UniqueName: \"kubernetes.io/projected/6ea32444-b868-4322-81b3-4fd19c09387b-kube-api-access-pvtm8\") pod \"6ea32444-b868-4322-81b3-4fd19c09387b\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") "
Nov 03 17:00:22.181667 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.181659    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-home\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-home\") pod \"6ea32444-b868-4322-81b3-4fd19c09387b\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") "
Nov 03 17:00:22.181831 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.181686    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-model-cache\") pod \"6ea32444-b868-4322-81b3-4fd19c09387b\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") "
Nov 03 17:00:22.181831 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.181813    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-dshm\") pod \"6ea32444-b868-4322-81b3-4fd19c09387b\" (UID: \"6ea32444-b868-4322-81b3-4fd19c09387b\") "
Nov 03 17:00:22.182227 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.182025    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-home" pod "6ea32444-b868-4322-81b3-4fd19c09387b" (UID: "6ea32444-b868-4322-81b3-4fd19c09387b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:00:22.182227 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.182176    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-model-cache" pod "6ea32444-b868-4322-81b3-4fd19c09387b" (UID: "6ea32444-b868-4322-81b3-4fd19c09387b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:00:22.182422 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.182293    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:22.182471 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.182431    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-tmp-dir" pod "6ea32444-b868-4322-81b3-4fd19c09387b" (UID: "6ea32444-b868-4322-81b3-4fd19c09387b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:00:22.184566 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.184536    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea32444-b868-4322-81b3-4fd19c09387b-tls-certs" pod "6ea32444-b868-4322-81b3-4fd19c09387b" (UID: "6ea32444-b868-4322-81b3-4fd19c09387b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:00:22.184788 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.184763    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea32444-b868-4322-81b3-4fd19c09387b-kube-api-access-pvtm8" pod "6ea32444-b868-4322-81b3-4fd19c09387b" (UID: "6ea32444-b868-4322-81b3-4fd19c09387b"). InnerVolumeSpecName "kube-api-access-pvtm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:00:22.184919 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.184893    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-dshm" pod "6ea32444-b868-4322-81b3-4fd19c09387b" (UID: "6ea32444-b868-4322-81b3-4fd19c09387b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:00:22.283391 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.283349    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:22.283391 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.283389    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea32444-b868-4322-81b3-4fd19c09387b-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:22.283625 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.283403    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:22.283625 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.283417    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvtm8\" (UniqueName: \"kubernetes.io/projected/6ea32444-b868-4322-81b3-4fd19c09387b-kube-api-access-pvtm8\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:22.283625 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.283433    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea32444-b868-4322-81b3-4fd19c09387b-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:22.550983 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.550943    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" path="/var/lib/kubelet/pods/b2917f91-5099-4d4b-9e7e-2d7c95ebb38f/volumes"
Nov 03 17:00:22.668341 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.668291    5112 generic.go:356] "Generic (PLEG): container finished" podID="6ea32444-b868-4322-81b3-4fd19c09387b" containerID="48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e" exitCode=0
Nov 03 17:00:22.668501 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.668362    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"
Nov 03 17:00:22.668569 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.668502    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z" event={"ID":"6ea32444-b868-4322-81b3-4fd19c09387b","Type":"ContainerDied","Data":"48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e"}
Nov 03 17:00:22.668569 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.668537    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z" event={"ID":"6ea32444-b868-4322-81b3-4fd19c09387b","Type":"ContainerDied","Data":"d721cad30145ba5e9196ae6afa09740e4473e47666c470d954bac5e5eaff9226"}
Nov 03 17:00:22.668569 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.668561    5112 scope.go:122] "RemoveContainer" containerID="48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e"
Nov 03 17:00:22.680420 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.680397    5112 scope.go:122] "RemoveContainer" containerID="48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e"
Nov 03 17:00:22.680705 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:22.680683    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e\": container with ID starting with 48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e not found: ID does not exist" containerID="48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e"
Nov 03 17:00:22.680783 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.680716    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e"} err="failed to get container status \"48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e\": rpc error: code = NotFound desc = could not find container \"48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e\": container with ID starting with 48f7f8c2ebdff7fef839c1fea6eebf0c37dde4ed6cb33dc97ce61f8e4eddc86e not found: ID does not exist"
Nov 03 17:00:22.829618 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.829534    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"]
Nov 03 17:00:22.876513 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:22.876478    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-963a2f/test-model-963a2f-kserve-6454954665-gft4z"]
Nov 03 17:00:24.550493 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:24.550463    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea32444-b868-4322-81b3-4fd19c09387b" path="/var/lib/kubelet/pods/6ea32444-b868-4322-81b3-4fd19c09387b/volumes"
Nov 03 17:00:37.882623 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:37.882582    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:00:39.318848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:39.318815    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:00:39.330227 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:39.330181    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:00:39.454632 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:39.454601    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:00:39.465303 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:39.465278    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:00:44.348492 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.348455    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"]
Nov 03 17:00:44.348923 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.348716    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz" podUID="8648b34b-c21b-49e6-b1d9-4b12b8df2c24" containerName="maas-api" containerID="cri-o://8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8" gracePeriod=30
Nov 03 17:00:44.624977 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.624947    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 17:00:44.646078 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.646039    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-kube-api-access-swmf2\" (UniqueName: \"kubernetes.io/projected/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-kube-api-access-swmf2\") pod \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") "
Nov 03 17:00:44.646265 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.646095    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-openshift-service-ca\") pod \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") "
Nov 03 17:00:44.646265 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.646129    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-metrics-certs\") pod \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") "
Nov 03 17:00:44.646265 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.646157    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls\") pod \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\" (UID: \"8648b34b-c21b-49e6-b1d9-4b12b8df2c24\") "
Nov 03 17:00:44.646616 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.646577    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-openshift-service-ca" pod "8648b34b-c21b-49e6-b1d9-4b12b8df2c24" (UID: "8648b34b-c21b-49e6-b1d9-4b12b8df2c24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:00:44.646759 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.646737    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:44.648526 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.648499    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-metrics-certs" pod "8648b34b-c21b-49e6-b1d9-4b12b8df2c24" (UID: "8648b34b-c21b-49e6-b1d9-4b12b8df2c24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:00:44.648630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.648538    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-kube-api-access-swmf2" pod "8648b34b-c21b-49e6-b1d9-4b12b8df2c24" (UID: "8648b34b-c21b-49e6-b1d9-4b12b8df2c24"). InnerVolumeSpecName "kube-api-access-swmf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:00:44.648821 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.648801    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls" pod "8648b34b-c21b-49e6-b1d9-4b12b8df2c24" (UID: "8648b34b-c21b-49e6-b1d9-4b12b8df2c24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:00:44.750414 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.747900    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmf2\" (UniqueName: \"kubernetes.io/projected/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-kube-api-access-swmf2\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:44.750414 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.747939    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:44.750414 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.747954    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8648b34b-c21b-49e6-b1d9-4b12b8df2c24-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:44.813048 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.813010    5112 generic.go:356] "Generic (PLEG): container finished" podID="8648b34b-c21b-49e6-b1d9-4b12b8df2c24" containerID="8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8" exitCode=0
Nov 03 17:00:44.813246 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.813053    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"
Nov 03 17:00:44.813246 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.813089    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz" event={"ID":"8648b34b-c21b-49e6-b1d9-4b12b8df2c24","Type":"ContainerDied","Data":"8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8"}
Nov 03 17:00:44.813246 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.813144    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz" event={"ID":"8648b34b-c21b-49e6-b1d9-4b12b8df2c24","Type":"ContainerDied","Data":"676fbd0de0ac76fda758bc34eb3a44b14bca06f6ab9e65f935415fc6ed040767"}
Nov 03 17:00:44.813246 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.813170    5112 scope.go:122] "RemoveContainer" containerID="8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8"
Nov 03 17:00:44.825112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.825088    5112 scope.go:122] "RemoveContainer" containerID="8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8"
Nov 03 17:00:44.825512 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:00:44.825483    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8\": container with ID starting with 8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8 not found: ID does not exist" containerID="8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8"
Nov 03 17:00:44.825618 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.825514    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8"} err="failed to get container status \"8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8\": rpc error: code = NotFound desc = could not find container \"8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8\": container with ID starting with 8cd9820c681982faff3df35b412b34bcd0a67bdb46bae161ee9fb846b7d41ee8 not found: ID does not exist"
Nov 03 17:00:44.974736 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.974699    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"]
Nov 03 17:00:44.980416 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:44.980388    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf492-78447bb465-wz9vz"]
Nov 03 17:00:45.512877 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.512836    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"]
Nov 03 17:00:45.513366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.513104    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz" podUID="8d628a4f-6ead-4e79-b7cd-57a2806f2e21" containerName="maas-api" containerID="cri-o://a35f6ed4ee35f7253d9096ad276fad7e42ea5435b77ce74cf8651089eeccf950" gracePeriod=30
Nov 03 17:00:45.823956 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.823920    5112 generic.go:356] "Generic (PLEG): container finished" podID="8d628a4f-6ead-4e79-b7cd-57a2806f2e21" containerID="a35f6ed4ee35f7253d9096ad276fad7e42ea5435b77ce74cf8651089eeccf950" exitCode=0
Nov 03 17:00:45.824226 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.824088    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz" event={"ID":"8d628a4f-6ead-4e79-b7cd-57a2806f2e21","Type":"ContainerDied","Data":"a35f6ed4ee35f7253d9096ad276fad7e42ea5435b77ce74cf8651089eeccf950"}
Nov 03 17:00:45.824226 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.824145    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz" event={"ID":"8d628a4f-6ead-4e79-b7cd-57a2806f2e21","Type":"ContainerDied","Data":"6ac741f4e2b1d72b037c175ea9890966d1956c0ae02e56fdd9068c0b508a23ef"}
Nov 03 17:00:45.824226 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.824164    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ac741f4e2b1d72b037c175ea9890966d1956c0ae02e56fdd9068c0b508a23ef"
Nov 03 17:00:45.824425 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.824301    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 17:00:45.862664 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.862624    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-metrics-certs\") pod \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") "
Nov 03 17:00:45.862853 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.862707    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls\") pod \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") "
Nov 03 17:00:45.862853 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.862828    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-kube-api-access-2c9t9\" (UniqueName: \"kubernetes.io/projected/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-kube-api-access-2c9t9\") pod \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") "
Nov 03 17:00:45.862974 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.862936    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-openshift-service-ca\") pod \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\" (UID: \"8d628a4f-6ead-4e79-b7cd-57a2806f2e21\") "
Nov 03 17:00:45.863380 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.863345    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-openshift-service-ca" pod "8d628a4f-6ead-4e79-b7cd-57a2806f2e21" (UID: "8d628a4f-6ead-4e79-b7cd-57a2806f2e21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:00:45.863716 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.863692    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:45.865355 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.865329    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls" pod "8d628a4f-6ead-4e79-b7cd-57a2806f2e21" (UID: "8d628a4f-6ead-4e79-b7cd-57a2806f2e21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:00:45.865463 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.865384    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-metrics-certs" pod "8d628a4f-6ead-4e79-b7cd-57a2806f2e21" (UID: "8d628a4f-6ead-4e79-b7cd-57a2806f2e21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:00:45.865656 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.865624    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-kube-api-access-2c9t9" pod "8d628a4f-6ead-4e79-b7cd-57a2806f2e21" (UID: "8d628a4f-6ead-4e79-b7cd-57a2806f2e21"). InnerVolumeSpecName "kube-api-access-2c9t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:00:45.965383 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.965336    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:45.965383 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.965375    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c9t9\" (UniqueName: \"kubernetes.io/projected/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-kube-api-access-2c9t9\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:45.965383 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:45.965393    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d628a4f-6ead-4e79-b7cd-57a2806f2e21-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:00:46.550332 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:46.550298    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8648b34b-c21b-49e6-b1d9-4b12b8df2c24" path="/var/lib/kubelet/pods/8648b34b-c21b-49e6-b1d9-4b12b8df2c24/volumes"
Nov 03 17:00:46.831652 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:46.831574    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"
Nov 03 17:00:46.993893 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:46.993858    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"]
Nov 03 17:00:47.000673 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:47.000640    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61be78-69df9b9d84-nzvvz"]
Nov 03 17:00:48.549547 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:00:48.549509    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d628a4f-6ead-4e79-b7cd-57a2806f2e21" path="/var/lib/kubelet/pods/8d628a4f-6ead-4e79-b7cd-57a2806f2e21/volumes"
Nov 03 17:01:02.019638 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:02.019595    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"]
Nov 03 17:01:07.883300 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:01:07.883263    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:01:09.285169 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:09.285130    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"]
Nov 03 17:01:14.286193 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.286144    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w" podUID="2075f8cb-c6c3-4c64-ac44-f153d43aeb70" containerName="payload-pre-processing" containerID="cri-o://1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c" gracePeriod=25
Nov 03 17:01:14.562245 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.562183    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 17:01:14.602609 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.602570    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-plugins-config-volume\") pod \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") "
Nov 03 17:01:14.602823 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.602634    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-kube-api-access-8rplv\" (UniqueName: \"kubernetes.io/projected/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-kube-api-access-8rplv\") pod \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\" (UID: \"2075f8cb-c6c3-4c64-ac44-f153d43aeb70\") "
Nov 03 17:01:14.603066 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.603032    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-plugins-config-volume" pod "2075f8cb-c6c3-4c64-ac44-f153d43aeb70" (UID: "2075f8cb-c6c3-4c64-ac44-f153d43aeb70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:01:14.603248 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.603185    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:14.605007 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.604978    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-kube-api-access-8rplv" pod "2075f8cb-c6c3-4c64-ac44-f153d43aeb70" (UID: "2075f8cb-c6c3-4c64-ac44-f153d43aeb70"). InnerVolumeSpecName "kube-api-access-8rplv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:01:14.704284 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:14.704239    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rplv\" (UniqueName: \"kubernetes.io/projected/2075f8cb-c6c3-4c64-ac44-f153d43aeb70-kube-api-access-8rplv\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:15.026857 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.026814    5112 generic.go:356] "Generic (PLEG): container finished" podID="2075f8cb-c6c3-4c64-ac44-f153d43aeb70" containerID="1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c" exitCode=0
Nov 03 17:01:15.027052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.026867    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w" event={"ID":"2075f8cb-c6c3-4c64-ac44-f153d43aeb70","Type":"ContainerDied","Data":"1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c"}
Nov 03 17:01:15.027052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.026917    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w" event={"ID":"2075f8cb-c6c3-4c64-ac44-f153d43aeb70","Type":"ContainerDied","Data":"fdf398a7c545d0df6dbe46a35351f53c6efdd25ec865f323ff8a640911960d79"}
Nov 03 17:01:15.027052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.026946    5112 scope.go:122] "RemoveContainer" containerID="1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c"
Nov 03 17:01:15.027052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.026978    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"
Nov 03 17:01:15.037435 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.037412    5112 scope.go:122] "RemoveContainer" containerID="1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c"
Nov 03 17:01:15.037772 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:01:15.037747    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c\": container with ID starting with 1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c not found: ID does not exist" containerID="1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c"
Nov 03 17:01:15.037852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.037781    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c"} err="failed to get container status \"1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c\": rpc error: code = NotFound desc = could not find container \"1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c\": container with ID starting with 1f7c1e2640e9fb8fa8366a8e2b5768d8c8de44089b0ddbb71b2176b50440596c not found: ID does not exist"
Nov 03 17:01:15.188933 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.188892    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"]
Nov 03 17:01:15.236505 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:15.236464    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-bcd756-6894748b59-9zn7w"]
Nov 03 17:01:16.549706 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:16.549671    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2075f8cb-c6c3-4c64-ac44-f153d43aeb70" path="/var/lib/kubelet/pods/2075f8cb-c6c3-4c64-ac44-f153d43aeb70/volumes"
Nov 03 17:01:17.040899 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.040845    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main" containerID="cri-o://548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23" gracePeriod=45
Nov 03 17:01:17.322422 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.322389    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 17:01:17.444426 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.444384    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-dshm\") pod \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") "
Nov 03 17:01:17.444634 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.444477    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tls-certs\" (UniqueName: \"kubernetes.io/secret/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tls-certs\") pod \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") "
Nov 03 17:01:17.444634 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.444614    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-home\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-home\") pod \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") "
Nov 03 17:01:17.444768 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.444690    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-kube-api-access-dld4s\" (UniqueName: \"kubernetes.io/projected/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-kube-api-access-dld4s\") pod \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") "
Nov 03 17:01:17.444768 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.444726    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-model-cache\") pod \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") "
Nov 03 17:01:17.444768 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.444764    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tmp-dir\") pod \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\" (UID: \"99609b93-a9c5-4641-a96d-2b7e7aa6baa1\") "
Nov 03 17:01:17.445373 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.445342    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tmp-dir" pod "99609b93-a9c5-4641-a96d-2b7e7aa6baa1" (UID: "99609b93-a9c5-4641-a96d-2b7e7aa6baa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:17.445574 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.445526    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-home" pod "99609b93-a9c5-4641-a96d-2b7e7aa6baa1" (UID: "99609b93-a9c5-4641-a96d-2b7e7aa6baa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:17.445802 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.445753    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-model-cache" pod "99609b93-a9c5-4641-a96d-2b7e7aa6baa1" (UID: "99609b93-a9c5-4641-a96d-2b7e7aa6baa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:17.448787 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.448755    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tls-certs" pod "99609b93-a9c5-4641-a96d-2b7e7aa6baa1" (UID: "99609b93-a9c5-4641-a96d-2b7e7aa6baa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:01:17.452764 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.452723    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-dshm" pod "99609b93-a9c5-4641-a96d-2b7e7aa6baa1" (UID: "99609b93-a9c5-4641-a96d-2b7e7aa6baa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:17.452922 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.452770    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-kube-api-access-dld4s" pod "99609b93-a9c5-4641-a96d-2b7e7aa6baa1" (UID: "99609b93-a9c5-4641-a96d-2b7e7aa6baa1"). InnerVolumeSpecName "kube-api-access-dld4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:01:17.546027 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.545986    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:17.546027 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.546025    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dld4s\" (UniqueName: \"kubernetes.io/projected/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-kube-api-access-dld4s\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:17.546027 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.546035    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:17.546298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.546046    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:17.546298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.546054    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:17.546298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:17.546062    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99609b93-a9c5-4641-a96d-2b7e7aa6baa1-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:18.048724 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.048682    5112 generic.go:356] "Generic (PLEG): container finished" podID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerID="548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23" exitCode=0
Nov 03 17:01:18.049238 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.048737    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"
Nov 03 17:01:18.049238 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.048754    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" event={"ID":"99609b93-a9c5-4641-a96d-2b7e7aa6baa1","Type":"ContainerDied","Data":"548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23"}
Nov 03 17:01:18.049238 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.048830    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w" event={"ID":"99609b93-a9c5-4641-a96d-2b7e7aa6baa1","Type":"ContainerDied","Data":"9ccfc5fc1e6c9772e336537904f1011e420d52491cf003009ad598e8c4c24e4b"}
Nov 03 17:01:18.049238 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.048854    5112 scope.go:122] "RemoveContainer" containerID="548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23"
Nov 03 17:01:18.062084 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.062058    5112 scope.go:122] "RemoveContainer" containerID="548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23"
Nov 03 17:01:18.062499 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:01:18.062477    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23\": container with ID starting with 548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23 not found: ID does not exist" containerID="548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23"
Nov 03 17:01:18.062581 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.062509    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23"} err="failed to get container status \"548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23\": rpc error: code = NotFound desc = could not find container \"548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23\": container with ID starting with 548d326273b64485687622827d277b9ca8d2a31f696dbacec94c500a2190ee23 not found: ID does not exist"
Nov 03 17:01:18.210616 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.210568    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"]
Nov 03 17:01:18.216456 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.216420    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bcd756/premium-w1-bcd756-kserve-7756bff6c4-pw69w"]
Nov 03 17:01:18.549414 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:18.549377    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" path="/var/lib/kubelet/pods/99609b93-a9c5-4641-a96d-2b7e7aa6baa1/volumes"
Nov 03 17:01:27.893570 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:27.893530    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"]
Nov 03 17:01:27.894033 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:27.893740    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk" podUID="acc9d684-66ba-485c-a926-deaa20d41fd6" containerName="maas-api" containerID="cri-o://9c7f2e5fe41391f88f5488ede56ea5bdfd656f5a59821196737ce6abd95a1413" gracePeriod=30
Nov 03 17:01:28.123383 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.123348    5112 generic.go:356] "Generic (PLEG): container finished" podID="acc9d684-66ba-485c-a926-deaa20d41fd6" containerID="9c7f2e5fe41391f88f5488ede56ea5bdfd656f5a59821196737ce6abd95a1413" exitCode=0
Nov 03 17:01:28.123582 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.123416    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk" event={"ID":"acc9d684-66ba-485c-a926-deaa20d41fd6","Type":"ContainerDied","Data":"9c7f2e5fe41391f88f5488ede56ea5bdfd656f5a59821196737ce6abd95a1413"}
Nov 03 17:01:28.170612 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.170575    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 17:01:28.270491 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.270459    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc9d684-66ba-485c-a926-deaa20d41fd6-kube-api-access-l8x62\" (UniqueName: \"kubernetes.io/projected/acc9d684-66ba-485c-a926-deaa20d41fd6-kube-api-access-l8x62\") pod \"acc9d684-66ba-485c-a926-deaa20d41fd6\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") "
Nov 03 17:01:28.270836 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.270710    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-metrics-certs\") pod \"acc9d684-66ba-485c-a926-deaa20d41fd6\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") "
Nov 03 17:01:28.270836 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.270759    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc9d684-66ba-485c-a926-deaa20d41fd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d684-66ba-485c-a926-deaa20d41fd6-openshift-service-ca\") pod \"acc9d684-66ba-485c-a926-deaa20d41fd6\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") "
Nov 03 17:01:28.270836 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.270787    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls\") pod \"acc9d684-66ba-485c-a926-deaa20d41fd6\" (UID: \"acc9d684-66ba-485c-a926-deaa20d41fd6\") "
Nov 03 17:01:28.271251 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.271211    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc9d684-66ba-485c-a926-deaa20d41fd6-openshift-service-ca" pod "acc9d684-66ba-485c-a926-deaa20d41fd6" (UID: "acc9d684-66ba-485c-a926-deaa20d41fd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:01:28.273104 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.273069    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc9d684-66ba-485c-a926-deaa20d41fd6-kube-api-access-l8x62" pod "acc9d684-66ba-485c-a926-deaa20d41fd6" (UID: "acc9d684-66ba-485c-a926-deaa20d41fd6"). InnerVolumeSpecName "kube-api-access-l8x62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:01:28.273104 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.273090    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-metrics-certs" pod "acc9d684-66ba-485c-a926-deaa20d41fd6" (UID: "acc9d684-66ba-485c-a926-deaa20d41fd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:01:28.273292 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.273116    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls" pod "acc9d684-66ba-485c-a926-deaa20d41fd6" (UID: "acc9d684-66ba-485c-a926-deaa20d41fd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:01:28.372403 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.372362    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:28.372403 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.372397    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc9d684-66ba-485c-a926-deaa20d41fd6-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:28.372403 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.372412    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc9d684-66ba-485c-a926-deaa20d41fd6-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:28.372665 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:28.372425    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8x62\" (UniqueName: \"kubernetes.io/projected/acc9d684-66ba-485c-a926-deaa20d41fd6-kube-api-access-l8x62\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:29.131595 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:29.131563    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"
Nov 03 17:01:29.131595 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:29.131568    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk" event={"ID":"acc9d684-66ba-485c-a926-deaa20d41fd6","Type":"ContainerDied","Data":"aa5ea1da46f5f86bdb472ba052fa69c503adf10824c370365ddb073d67512b0f"}
Nov 03 17:01:29.132165 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:29.131640    5112 scope.go:122] "RemoveContainer" containerID="9c7f2e5fe41391f88f5488ede56ea5bdfd656f5a59821196737ce6abd95a1413"
Nov 03 17:01:29.291892 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:29.291852    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"]
Nov 03 17:01:29.338939 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:29.338867    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-963a2f-6666b77556-qs9hk"]
Nov 03 17:01:30.549961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:30.549919    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc9d684-66ba-485c-a926-deaa20d41fd6" path="/var/lib/kubelet/pods/acc9d684-66ba-485c-a926-deaa20d41fd6/volumes"
Nov 03 17:01:37.885227 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:01:37.885149    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:01:40.126486 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:40.126452    5112 scope.go:122] "RemoveContainer" containerID="a35f6ed4ee35f7253d9096ad276fad7e42ea5435b77ce74cf8651089eeccf950"
Nov 03 17:01:43.750173 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:43.750135    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"]
Nov 03 17:01:43.750629 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:43.750366    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf" podUID="8052eca5-aedb-4c3d-b37f-051c24042490" containerName="maas-api" containerID="cri-o://2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258" gracePeriod=30
Nov 03 17:01:44.023144 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.023116    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 17:01:44.071460 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.071420    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8052eca5-aedb-4c3d-b37f-051c24042490-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052eca5-aedb-4c3d-b37f-051c24042490-openshift-service-ca\") pod \"8052eca5-aedb-4c3d-b37f-051c24042490\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") "
Nov 03 17:01:44.071682 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.071526    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-metrics-certs\") pod \"8052eca5-aedb-4c3d-b37f-051c24042490\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") "
Nov 03 17:01:44.071682 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.071578    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls\") pod \"8052eca5-aedb-4c3d-b37f-051c24042490\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") "
Nov 03 17:01:44.071682 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.071608    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8052eca5-aedb-4c3d-b37f-051c24042490-kube-api-access-cf48z\" (UniqueName: \"kubernetes.io/projected/8052eca5-aedb-4c3d-b37f-051c24042490-kube-api-access-cf48z\") pod \"8052eca5-aedb-4c3d-b37f-051c24042490\" (UID: \"8052eca5-aedb-4c3d-b37f-051c24042490\") "
Nov 03 17:01:44.071905 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.071874    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8052eca5-aedb-4c3d-b37f-051c24042490-openshift-service-ca" pod "8052eca5-aedb-4c3d-b37f-051c24042490" (UID: "8052eca5-aedb-4c3d-b37f-051c24042490"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:01:44.072074 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.072050    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052eca5-aedb-4c3d-b37f-051c24042490-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:44.073817 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.073783    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls" pod "8052eca5-aedb-4c3d-b37f-051c24042490" (UID: "8052eca5-aedb-4c3d-b37f-051c24042490"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:01:44.074300 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.074278    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8052eca5-aedb-4c3d-b37f-051c24042490-kube-api-access-cf48z" pod "8052eca5-aedb-4c3d-b37f-051c24042490" (UID: "8052eca5-aedb-4c3d-b37f-051c24042490"). InnerVolumeSpecName "kube-api-access-cf48z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:01:44.074788 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.074761    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-metrics-certs" pod "8052eca5-aedb-4c3d-b37f-051c24042490" (UID: "8052eca5-aedb-4c3d-b37f-051c24042490"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:01:44.173839 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.173798    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:44.173839 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.173840    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052eca5-aedb-4c3d-b37f-051c24042490-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:44.174062 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.173857    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf48z\" (UniqueName: \"kubernetes.io/projected/8052eca5-aedb-4c3d-b37f-051c24042490-kube-api-access-cf48z\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:44.231399 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.231356    5112 generic.go:356] "Generic (PLEG): container finished" podID="8052eca5-aedb-4c3d-b37f-051c24042490" containerID="2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258" exitCode=0
Nov 03 17:01:44.231606 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.231397    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf" event={"ID":"8052eca5-aedb-4c3d-b37f-051c24042490","Type":"ContainerDied","Data":"2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258"}
Nov 03 17:01:44.231606 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.231457    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"
Nov 03 17:01:44.231606 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.231476    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf" event={"ID":"8052eca5-aedb-4c3d-b37f-051c24042490","Type":"ContainerDied","Data":"519ccc2ecedcd1e801ff08cd80f99116848e5f3a028958c5065e80fc46b97eca"}
Nov 03 17:01:44.231606 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.231509    5112 scope.go:122] "RemoveContainer" containerID="2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258"
Nov 03 17:01:44.244112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.244087    5112 scope.go:122] "RemoveContainer" containerID="2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258"
Nov 03 17:01:44.244509 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:01:44.244487    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258\": container with ID starting with 2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258 not found: ID does not exist" containerID="2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258"
Nov 03 17:01:44.244574 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.244518    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258"} err="failed to get container status \"2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258\": rpc error: code = NotFound desc = could not find container \"2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258\": container with ID starting with 2c725860b5d8fa4c797600cc0afe28b7ca128de69c59607ee5b51f2c7b75c258 not found: ID does not exist"
Nov 03 17:01:44.410933 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.410901    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"]
Nov 03 17:01:44.420387 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.420355    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-bcd756-5c68b5948f-wdqmf"]
Nov 03 17:01:44.549381 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:44.549346    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8052eca5-aedb-4c3d-b37f-051c24042490" path="/var/lib/kubelet/pods/8052eca5-aedb-4c3d-b37f-051c24042490/volumes"
Nov 03 17:01:48.973878 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:48.973784    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"]
Nov 03 17:01:54.519393 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:54.519353    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"]
Nov 03 17:01:54.519831 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:54.519595    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy" containerID="cri-o://97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e" gracePeriod=30
Nov 03 17:01:56.786086 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:56.786045    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"]
Nov 03 17:01:56.886589 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:56.886552    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"]
Nov 03 17:01:59.791296 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.791223    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 17:01:59.949979 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.949941    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-token\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-token\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.949979 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.949992    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-kube-api-access-dxf8q\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-kube-api-access-dxf8q\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950010    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istiod-ca-cert\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950046    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-podinfo\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950091    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-data\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950113    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-envoy\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950212    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-certs\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950549 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950281    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-socket\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950549 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950355    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-credential-socket\") pod \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\" (UID: \"f7d861a3-b90a-456f-9f54-ff31f6c09f1e\") "
Nov 03 17:01:59.950549 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950435    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istiod-ca-cert" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:01:59.950744 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950723    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-socket" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:59.950804 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950783    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:01:59.950864 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950835    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-certs" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:59.951241 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950925    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-data" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:59.951241 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.950942    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-credential-socket" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:59.953672 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.953644    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-kube-api-access-dxf8q" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "kube-api-access-dxf8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:01:59.961038 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.961005    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-token" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:01:59.961221 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.961128    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-envoy" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:01:59.961294 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:01:59.961216    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-podinfo" pod "f7d861a3-b90a-456f-9f54-ff31f6c09f1e" (UID: "f7d861a3-b90a-456f-9f54-ff31f6c09f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052075    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052107    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxf8q\" (UniqueName: \"kubernetes.io/projected/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-kube-api-access-dxf8q\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052121    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052129    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052138    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052147    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052155    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.052163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.052163    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d861a3-b90a-456f-9f54-ff31f6c09f1e-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:00.335146 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.335056    5112 generic.go:356] "Generic (PLEG): container finished" podID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerID="97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e" exitCode=0
Nov 03 17:02:00.335146 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.335104    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"
Nov 03 17:02:00.335146 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.335136    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" event={"ID":"f7d861a3-b90a-456f-9f54-ff31f6c09f1e","Type":"ContainerDied","Data":"97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e"}
Nov 03 17:02:00.335542 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.335192    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv" event={"ID":"f7d861a3-b90a-456f-9f54-ff31f6c09f1e","Type":"ContainerDied","Data":"ee89b0a75c11466c420f7ad801aa248b7a679c039b96d2f36e5027d41ef16851"}
Nov 03 17:02:00.335542 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.335229    5112 scope.go:122] "RemoveContainer" containerID="97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e"
Nov 03 17:02:00.347600 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.347574    5112 scope.go:122] "RemoveContainer" containerID="97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e"
Nov 03 17:02:00.347935 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:00.347917    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e\": container with ID starting with 97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e not found: ID does not exist" containerID="97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e"
Nov 03 17:02:00.347986 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.347946    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e"} err="failed to get container status \"97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e\": rpc error: code = NotFound desc = could not find container \"97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e\": container with ID starting with 97c661403ba7bf428aeb54ae6f5a50701f92ffc78353d4974de8edafbb2e638e not found: ID does not exist"
Nov 03 17:02:00.508998 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.508960    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"]
Nov 03 17:02:00.557154 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:00.557118    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bcd756-openshift-default-fb4b6cdf-g29lv"]
Nov 03 17:02:01.786442 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:01.786378    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv" podUID="b0ff4aac-032d-483d-8ea3-648619a23e2d" containerName="payload-processing" containerID="cri-o://ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b" gracePeriod=25
Nov 03 17:02:01.887173 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:01.887088    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w" podUID="105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" containerName="payload-pre-processing" containerID="cri-o://6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9" gracePeriod=25
Nov 03 17:02:02.086270 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.085808    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 17:02:02.160255 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.160228    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 17:02:02.171355 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.171325    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-plugins-config-volume\") pod \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") "
Nov 03 17:02:02.171512 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.171440    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ff4aac-032d-483d-8ea3-648619a23e2d-kube-api-access-xp7nx\" (UniqueName: \"kubernetes.io/projected/b0ff4aac-032d-483d-8ea3-648619a23e2d-kube-api-access-xp7nx\") pod \"b0ff4aac-032d-483d-8ea3-648619a23e2d\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") "
Nov 03 17:02:02.171512 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.171475    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ff4aac-032d-483d-8ea3-648619a23e2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ff4aac-032d-483d-8ea3-648619a23e2d-plugins-config-volume\") pod \"b0ff4aac-032d-483d-8ea3-648619a23e2d\" (UID: \"b0ff4aac-032d-483d-8ea3-648619a23e2d\") "
Nov 03 17:02:02.171623 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.171522    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-kube-api-access-qsd8q\" (UniqueName: \"kubernetes.io/projected/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-kube-api-access-qsd8q\") pod \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\" (UID: \"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb\") "
Nov 03 17:02:02.171805 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.171783    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-plugins-config-volume" pod "105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" (UID: "105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:02:02.171910 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.171881    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ff4aac-032d-483d-8ea3-648619a23e2d-plugins-config-volume" pod "b0ff4aac-032d-483d-8ea3-648619a23e2d" (UID: "b0ff4aac-032d-483d-8ea3-648619a23e2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:02:02.171981 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.171945    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:02.173771 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.173740    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-kube-api-access-qsd8q" pod "105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" (UID: "105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb"). InnerVolumeSpecName "kube-api-access-qsd8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:02.173950 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.173928    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ff4aac-032d-483d-8ea3-648619a23e2d-kube-api-access-xp7nx" pod "b0ff4aac-032d-483d-8ea3-648619a23e2d" (UID: "b0ff4aac-032d-483d-8ea3-648619a23e2d"). InnerVolumeSpecName "kube-api-access-xp7nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:02.272758 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.272711    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ff4aac-032d-483d-8ea3-648619a23e2d-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:02.272758 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.272748    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsd8q\" (UniqueName: \"kubernetes.io/projected/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb-kube-api-access-qsd8q\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:02.272758 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.272761    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp7nx\" (UniqueName: \"kubernetes.io/projected/b0ff4aac-032d-483d-8ea3-648619a23e2d-kube-api-access-xp7nx\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:02.352362 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.352261    5112 generic.go:356] "Generic (PLEG): container finished" podID="105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" containerID="6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9" exitCode=0
Nov 03 17:02:02.352362 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.352306    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"
Nov 03 17:02:02.352362 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.352335    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w" event={"ID":"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb","Type":"ContainerDied","Data":"6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9"}
Nov 03 17:02:02.352652 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.352433    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w" event={"ID":"105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb","Type":"ContainerDied","Data":"bcd8a7cedb988fd8602987f4c8ee85562900da0b463b50b1f2bb2141f31373a4"}
Nov 03 17:02:02.352652 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.352465    5112 scope.go:122] "RemoveContainer" containerID="6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9"
Nov 03 17:02:02.354361 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.354321    5112 generic.go:356] "Generic (PLEG): container finished" podID="b0ff4aac-032d-483d-8ea3-648619a23e2d" containerID="ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b" exitCode=0
Nov 03 17:02:02.354500 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.354391    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"
Nov 03 17:02:02.354500 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.354410    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv" event={"ID":"b0ff4aac-032d-483d-8ea3-648619a23e2d","Type":"ContainerDied","Data":"ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b"}
Nov 03 17:02:02.354500 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.354454    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv" event={"ID":"b0ff4aac-032d-483d-8ea3-648619a23e2d","Type":"ContainerDied","Data":"52ad250d6c0e5ff53ee92a03d75c76feb5ec4f50cc753ff810c2cef83d8a71a9"}
Nov 03 17:02:02.375108 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.375073    5112 scope.go:122] "RemoveContainer" containerID="6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9"
Nov 03 17:02:02.376541 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:02.376507    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9\": container with ID starting with 6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9 not found: ID does not exist" containerID="6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9"
Nov 03 17:02:02.376668 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.376551    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9"} err="failed to get container status \"6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9\": rpc error: code = NotFound desc = could not find container \"6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9\": container with ID starting with 6d1e54924e8b0c5eb6d96c913b5edb3eebea3572d3be95ac7600005ad8673ab9 not found: ID does not exist"
Nov 03 17:02:02.376668 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.376581    5112 scope.go:122] "RemoveContainer" containerID="ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b"
Nov 03 17:02:02.387783 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.387761    5112 scope.go:122] "RemoveContainer" containerID="ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b"
Nov 03 17:02:02.388164 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:02.388139    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b\": container with ID starting with ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b not found: ID does not exist" containerID="ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b"
Nov 03 17:02:02.389061 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.388179    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b"} err="failed to get container status \"ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b\": rpc error: code = NotFound desc = could not find container \"ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b\": container with ID starting with ceacb04684e5f062e1f87aba2a0ddfcf9da84e55667534bb9cad64b788affb0b not found: ID does not exist"
Nov 03 17:02:02.517395 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.517356    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"]
Nov 03 17:02:02.549188 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.549154    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" path="/var/lib/kubelet/pods/f7d861a3-b90a-456f-9f54-ff31f6c09f1e/volumes"
Nov 03 17:02:02.556042 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.556006    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42da0f-d68f8696c-5x25w"]
Nov 03 17:02:02.665646 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.665605    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"]
Nov 03 17:02:02.674599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:02.674564    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42da0f-b59659bdf-kt2zv"]
Nov 03 17:02:04.000303 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.000257    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main" containerID="cri-o://551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34" gracePeriod=45
Nov 03 17:02:04.275323 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.275294    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 17:02:04.293440 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293404    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-home\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-home\") pod \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") "
Nov 03 17:02:04.293630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293448    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tls-certs\") pod \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") "
Nov 03 17:02:04.293630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293498    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-model-cache\") pod \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") "
Nov 03 17:02:04.293630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293545    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-kube-api-access-rjnd5\" (UniqueName: \"kubernetes.io/projected/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-kube-api-access-rjnd5\") pod \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") "
Nov 03 17:02:04.293630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293584    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tmp-dir\") pod \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") "
Nov 03 17:02:04.293859 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293630    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-dshm\") pod \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\" (UID: \"a7af5ace-a5cd-4b3b-b88b-61969a16ea82\") "
Nov 03 17:02:04.293859 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293686    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-home" pod "a7af5ace-a5cd-4b3b-b88b-61969a16ea82" (UID: "a7af5ace-a5cd-4b3b-b88b-61969a16ea82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:04.293859 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293822    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-model-cache" pod "a7af5ace-a5cd-4b3b-b88b-61969a16ea82" (UID: "a7af5ace-a5cd-4b3b-b88b-61969a16ea82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:04.294003 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293858    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tmp-dir" pod "a7af5ace-a5cd-4b3b-b88b-61969a16ea82" (UID: "a7af5ace-a5cd-4b3b-b88b-61969a16ea82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:04.294003 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.293991    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:04.294067 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.294010    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:04.294067 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.294024    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:04.295962 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.295939    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tls-certs" pod "a7af5ace-a5cd-4b3b-b88b-61969a16ea82" (UID: "a7af5ace-a5cd-4b3b-b88b-61969a16ea82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:02:04.296065 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.295967    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-dshm" pod "a7af5ace-a5cd-4b3b-b88b-61969a16ea82" (UID: "a7af5ace-a5cd-4b3b-b88b-61969a16ea82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:04.296065 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.295974    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-kube-api-access-rjnd5" pod "a7af5ace-a5cd-4b3b-b88b-61969a16ea82" (UID: "a7af5ace-a5cd-4b3b-b88b-61969a16ea82"). InnerVolumeSpecName "kube-api-access-rjnd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:04.369683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.369643    5112 generic.go:356] "Generic (PLEG): container finished" podID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerID="551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34" exitCode=0
Nov 03 17:02:04.369873 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.369701    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"
Nov 03 17:02:04.369873 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.369731    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" event={"ID":"a7af5ace-a5cd-4b3b-b88b-61969a16ea82","Type":"ContainerDied","Data":"551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34"}
Nov 03 17:02:04.369873 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.369775    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv" event={"ID":"a7af5ace-a5cd-4b3b-b88b-61969a16ea82","Type":"ContainerDied","Data":"4a902c3eabd28c859ecd7eaead81dcc8da56efcb2ff1a7317bfa6ae430f89143"}
Nov 03 17:02:04.369873 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.369794    5112 scope.go:122] "RemoveContainer" containerID="551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34"
Nov 03 17:02:04.380904 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.380882    5112 scope.go:122] "RemoveContainer" containerID="551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34"
Nov 03 17:02:04.381250 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:04.381226    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34\": container with ID starting with 551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34 not found: ID does not exist" containerID="551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34"
Nov 03 17:02:04.381314 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.381259    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34"} err="failed to get container status \"551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34\": rpc error: code = NotFound desc = could not find container \"551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34\": container with ID starting with 551e1a5dff05dd51f4894ac6802d50ce29b1f08678e9822a9d331126ecab0a34 not found: ID does not exist"
Nov 03 17:02:04.394874 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.394842    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnd5\" (UniqueName: \"kubernetes.io/projected/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-kube-api-access-rjnd5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:04.394874 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.394871    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:04.394874 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.394881    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7af5ace-a5cd-4b3b-b88b-61969a16ea82-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:04.500094 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.500059    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"]
Nov 03 17:02:04.502654 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.502629    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42da0f/distinct-2-w0-42da0f-kserve-7b49f87fc7-dpgwv"]
Nov 03 17:02:04.549093 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.549009    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" path="/var/lib/kubelet/pods/105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb/volumes"
Nov 03 17:02:04.549477 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.549463    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" path="/var/lib/kubelet/pods/a7af5ace-a5cd-4b3b-b88b-61969a16ea82/volumes"
Nov 03 17:02:04.549811 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:04.549798    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ff4aac-032d-483d-8ea3-648619a23e2d" path="/var/lib/kubelet/pods/b0ff4aac-032d-483d-8ea3-648619a23e2d/volumes"
Nov 03 17:02:07.882997 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:07.882957    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:02:22.024734 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.024686    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"]
Nov 03 17:02:22.025382 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.024979    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz" podUID="2478e2ce-c49e-483d-9ce4-60f0283a323e" containerName="maas-api" containerID="cri-o://9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411" gracePeriod=30
Nov 03 17:02:22.293417 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.293349    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 17:02:22.403185 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.403148    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2478e2ce-c49e-483d-9ce4-60f0283a323e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2478e2ce-c49e-483d-9ce4-60f0283a323e-openshift-service-ca\") pod \"2478e2ce-c49e-483d-9ce4-60f0283a323e\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") "
Nov 03 17:02:22.403383 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.403229    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls\") pod \"2478e2ce-c49e-483d-9ce4-60f0283a323e\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") "
Nov 03 17:02:22.403383 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.403350    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2478e2ce-c49e-483d-9ce4-60f0283a323e-kube-api-access-g7pp5\" (UniqueName: \"kubernetes.io/projected/2478e2ce-c49e-483d-9ce4-60f0283a323e-kube-api-access-g7pp5\") pod \"2478e2ce-c49e-483d-9ce4-60f0283a323e\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") "
Nov 03 17:02:22.403514 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.403449    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-metrics-certs\") pod \"2478e2ce-c49e-483d-9ce4-60f0283a323e\" (UID: \"2478e2ce-c49e-483d-9ce4-60f0283a323e\") "
Nov 03 17:02:22.403634 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.403609    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2478e2ce-c49e-483d-9ce4-60f0283a323e-openshift-service-ca" pod "2478e2ce-c49e-483d-9ce4-60f0283a323e" (UID: "2478e2ce-c49e-483d-9ce4-60f0283a323e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:02:22.403829 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.403806    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2478e2ce-c49e-483d-9ce4-60f0283a323e-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:22.405486 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.405455    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls" pod "2478e2ce-c49e-483d-9ce4-60f0283a323e" (UID: "2478e2ce-c49e-483d-9ce4-60f0283a323e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:02:22.405594 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.405506    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2478e2ce-c49e-483d-9ce4-60f0283a323e-kube-api-access-g7pp5" pod "2478e2ce-c49e-483d-9ce4-60f0283a323e" (UID: "2478e2ce-c49e-483d-9ce4-60f0283a323e"). InnerVolumeSpecName "kube-api-access-g7pp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:22.405594 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.405573    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-metrics-certs" pod "2478e2ce-c49e-483d-9ce4-60f0283a323e" (UID: "2478e2ce-c49e-483d-9ce4-60f0283a323e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:02:22.487086 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.487044    5112 generic.go:356] "Generic (PLEG): container finished" podID="2478e2ce-c49e-483d-9ce4-60f0283a323e" containerID="9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411" exitCode=0
Nov 03 17:02:22.487284 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.487089    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"
Nov 03 17:02:22.487284 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.487124    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz" event={"ID":"2478e2ce-c49e-483d-9ce4-60f0283a323e","Type":"ContainerDied","Data":"9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411"}
Nov 03 17:02:22.487284 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.487186    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz" event={"ID":"2478e2ce-c49e-483d-9ce4-60f0283a323e","Type":"ContainerDied","Data":"a8119960024f9f3f13571b0b3dd81c94af5ccac9acc511f73096410bd184b5f7"}
Nov 03 17:02:22.487284 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.487242    5112 scope.go:122] "RemoveContainer" containerID="9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411"
Nov 03 17:02:22.498727 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.498706    5112 scope.go:122] "RemoveContainer" containerID="9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411"
Nov 03 17:02:22.499028 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:22.499010    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411\": container with ID starting with 9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411 not found: ID does not exist" containerID="9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411"
Nov 03 17:02:22.499104 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.499040    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411"} err="failed to get container status \"9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411\": rpc error: code = NotFound desc = could not find container \"9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411\": container with ID starting with 9c9819c2e6d9c1e3e1fd7a72b96a75556e4f01e644b00057e3babab4fcb68411 not found: ID does not exist"
Nov 03 17:02:22.504657 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.504627    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7pp5\" (UniqueName: \"kubernetes.io/projected/2478e2ce-c49e-483d-9ce4-60f0283a323e-kube-api-access-g7pp5\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:22.504657 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.504654    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:22.504806 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.504664    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2478e2ce-c49e-483d-9ce4-60f0283a323e-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:22.647471 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.647430    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"]
Nov 03 17:02:22.649937 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:22.649916    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-509f14-79b9df6bc6-sb6xz"]
Nov 03 17:02:24.549460 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:24.549425    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2478e2ce-c49e-483d-9ce4-60f0283a323e" path="/var/lib/kubelet/pods/2478e2ce-c49e-483d-9ce4-60f0283a323e/volumes"
Nov 03 17:02:32.836733 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:32.836698    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"]
Nov 03 17:02:32.837166 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:32.836898    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" containerID="cri-o://2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729" gracePeriod=30
Nov 03 17:02:36.737845 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:36.737804    5112 patch_prober.go:29] interesting pod/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:02:36.738279 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:36.737882    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:02:37.883108 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:37.883073    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:02:38.093296 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.093272    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 17:02:38.155866 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.155838    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-socket\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156034 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.155911    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-credential-socket\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156034 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.155939    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-data\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156034 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156018    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-envoy\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156181 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156053    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-podinfo\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156181 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156088    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-kube-api-access-whj5h\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-kube-api-access-whj5h\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156181 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156122    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-token\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-token\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156181 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156161    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-socket" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:38.156421 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156168    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3fbeb18-4414-43b8-a190-36beb56e4193-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fbeb18-4414-43b8-a190-36beb56e4193-istiod-ca-cert\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156421 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156288    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-credential-socket" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:38.156421 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156321    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-certs\") pod \"c3fbeb18-4414-43b8-a190-36beb56e4193\" (UID: \"c3fbeb18-4414-43b8-a190-36beb56e4193\") "
Nov 03 17:02:38.156571 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156426    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-data" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:38.156571 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156499    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fbeb18-4414-43b8-a190-36beb56e4193-istiod-ca-cert" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:02:38.156668 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156597    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-certs" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:38.156777 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156762    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.156835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156780    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.156835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156790    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.156835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156800    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fbeb18-4414-43b8-a190-36beb56e4193-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.156835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.156809    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.158560 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.158516    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-podinfo" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:02:38.158710 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.158605    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-envoy" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:38.158710 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.158669    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-kube-api-access-whj5h" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "kube-api-access-whj5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:38.158838 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.158820    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-token" pod "c3fbeb18-4414-43b8-a190-36beb56e4193" (UID: "c3fbeb18-4414-43b8-a190-36beb56e4193"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:38.257855 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.257823    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.257855 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.257854    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.258076 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.257870    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fbeb18-4414-43b8-a190-36beb56e4193-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.258076 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.257883    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whj5h\" (UniqueName: \"kubernetes.io/projected/c3fbeb18-4414-43b8-a190-36beb56e4193-kube-api-access-whj5h\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:38.580626 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.580596    5112 generic.go:356] "Generic (PLEG): container finished" podID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerID="2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729" exitCode=0
Nov 03 17:02:38.580801 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.580645    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"
Nov 03 17:02:38.580801 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.580677    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" event={"ID":"c3fbeb18-4414-43b8-a190-36beb56e4193","Type":"ContainerDied","Data":"2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729"}
Nov 03 17:02:38.580801 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.580729    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn" event={"ID":"c3fbeb18-4414-43b8-a190-36beb56e4193","Type":"ContainerDied","Data":"3b238b5fa2def0d04e69a3abeec5bd5de66c401f781526eaf9e106c347951a5f"}
Nov 03 17:02:38.580801 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.580761    5112 scope.go:122] "RemoveContainer" containerID="2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729"
Nov 03 17:02:38.590386 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.590221    5112 scope.go:122] "RemoveContainer" containerID="2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729"
Nov 03 17:02:38.590621 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:38.590576    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729\": container with ID starting with 2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729 not found: ID does not exist" containerID="2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729"
Nov 03 17:02:38.590704 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.590632    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729"} err="failed to get container status \"2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729\": rpc error: code = NotFound desc = could not find container \"2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729\": container with ID starting with 2cb87e6b82c6e28584045f308fe98c68d815bbb3c337ed106eef697db9eed729 not found: ID does not exist"
Nov 03 17:02:38.742805 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.742762    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"]
Nov 03 17:02:38.794043 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:38.793997    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42da0f-openshift-default-69bcdf78d7-68nwn"]
Nov 03 17:02:39.791370 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:39.791330    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"]
Nov 03 17:02:40.550413 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:40.550333    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" path="/var/lib/kubelet/pods/c3fbeb18-4414-43b8-a190-36beb56e4193/volumes"
Nov 03 17:02:46.856250 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:46.856216    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"]
Nov 03 17:02:46.957002 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:46.956964    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"]
Nov 03 17:02:51.857490 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:51.857434    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b" podUID="2c6e4908-9236-4452-a3e1-3f26a5fc0449" containerName="payload-processing" containerID="cri-o://da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478" gracePeriod=25
Nov 03 17:02:51.957437 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:51.957365    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz" podUID="e5229c4c-b2fb-4418-ab4c-89458192001e" containerName="payload-pre-processing" containerID="cri-o://bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff" gracePeriod=25
Nov 03 17:02:52.118504 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.118436    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 17:02:52.198376 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.198342    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6e4908-9236-4452-a3e1-3f26a5fc0449-kube-api-access-h7p2c\" (UniqueName: \"kubernetes.io/projected/2c6e4908-9236-4452-a3e1-3f26a5fc0449-kube-api-access-h7p2c\") pod \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") "
Nov 03 17:02:52.198562 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.198426    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6e4908-9236-4452-a3e1-3f26a5fc0449-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e4908-9236-4452-a3e1-3f26a5fc0449-plugins-config-volume\") pod \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\" (UID: \"2c6e4908-9236-4452-a3e1-3f26a5fc0449\") "
Nov 03 17:02:52.198790 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.198760    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6e4908-9236-4452-a3e1-3f26a5fc0449-plugins-config-volume" pod "2c6e4908-9236-4452-a3e1-3f26a5fc0449" (UID: "2c6e4908-9236-4452-a3e1-3f26a5fc0449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:02:52.200788 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.200756    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6e4908-9236-4452-a3e1-3f26a5fc0449-kube-api-access-h7p2c" pod "2c6e4908-9236-4452-a3e1-3f26a5fc0449" (UID: "2c6e4908-9236-4452-a3e1-3f26a5fc0449"). InnerVolumeSpecName "kube-api-access-h7p2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:52.217277 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.217253    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 17:02:52.299219 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.299159    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5229c4c-b2fb-4418-ab4c-89458192001e-kube-api-access-zzcgf\" (UniqueName: \"kubernetes.io/projected/e5229c4c-b2fb-4418-ab4c-89458192001e-kube-api-access-zzcgf\") pod \"e5229c4c-b2fb-4418-ab4c-89458192001e\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") "
Nov 03 17:02:52.299399 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.299309    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5229c4c-b2fb-4418-ab4c-89458192001e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5229c4c-b2fb-4418-ab4c-89458192001e-plugins-config-volume\") pod \"e5229c4c-b2fb-4418-ab4c-89458192001e\" (UID: \"e5229c4c-b2fb-4418-ab4c-89458192001e\") "
Nov 03 17:02:52.299561 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.299546    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7p2c\" (UniqueName: \"kubernetes.io/projected/2c6e4908-9236-4452-a3e1-3f26a5fc0449-kube-api-access-h7p2c\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:52.299600 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.299563    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e4908-9236-4452-a3e1-3f26a5fc0449-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:52.299721 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.299688    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5229c4c-b2fb-4418-ab4c-89458192001e-plugins-config-volume" pod "e5229c4c-b2fb-4418-ab4c-89458192001e" (UID: "e5229c4c-b2fb-4418-ab4c-89458192001e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:02:52.301469 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.301449    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5229c4c-b2fb-4418-ab4c-89458192001e-kube-api-access-zzcgf" pod "e5229c4c-b2fb-4418-ab4c-89458192001e" (UID: "e5229c4c-b2fb-4418-ab4c-89458192001e"). InnerVolumeSpecName "kube-api-access-zzcgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:52.400778 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.400743    5112 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5229c4c-b2fb-4418-ab4c-89458192001e-plugins-config-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:52.400778 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.400775    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzcgf\" (UniqueName: \"kubernetes.io/projected/e5229c4c-b2fb-4418-ab4c-89458192001e-kube-api-access-zzcgf\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:52.658839 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.658752    5112 generic.go:356] "Generic (PLEG): container finished" podID="2c6e4908-9236-4452-a3e1-3f26a5fc0449" containerID="da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478" exitCode=0
Nov 03 17:02:52.658839 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.658806    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"
Nov 03 17:02:52.659092 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.658837    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b" event={"ID":"2c6e4908-9236-4452-a3e1-3f26a5fc0449","Type":"ContainerDied","Data":"da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478"}
Nov 03 17:02:52.659092 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.658904    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b" event={"ID":"2c6e4908-9236-4452-a3e1-3f26a5fc0449","Type":"ContainerDied","Data":"38ea8f04bda53551ce91a11b463ed7f82d43eae941f82e6726f56ed3d412c5ca"}
Nov 03 17:02:52.659092 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.658935    5112 scope.go:122] "RemoveContainer" containerID="da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478"
Nov 03 17:02:52.660524 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.660498    5112 generic.go:356] "Generic (PLEG): container finished" podID="e5229c4c-b2fb-4418-ab4c-89458192001e" containerID="bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff" exitCode=0
Nov 03 17:02:52.660663 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.660553    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"
Nov 03 17:02:52.660663 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.660588    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz" event={"ID":"e5229c4c-b2fb-4418-ab4c-89458192001e","Type":"ContainerDied","Data":"bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff"}
Nov 03 17:02:52.660663 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.660623    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz" event={"ID":"e5229c4c-b2fb-4418-ab4c-89458192001e","Type":"ContainerDied","Data":"6786561f02672e12e4efcba9926772dcdb13b1f2da126cdfb0aa61a1a9549021"}
Nov 03 17:02:52.669774 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.669754    5112 scope.go:122] "RemoveContainer" containerID="da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478"
Nov 03 17:02:52.670095 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:52.670074    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478\": container with ID starting with da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478 not found: ID does not exist" containerID="da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478"
Nov 03 17:02:52.670142 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.670105    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478"} err="failed to get container status \"da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478\": rpc error: code = NotFound desc = could not find container \"da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478\": container with ID starting with da0fa22154780304fea770bf4baa5a418194b672d21c08f9472501feb5627478 not found: ID does not exist"
Nov 03 17:02:52.670142 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.670124    5112 scope.go:122] "RemoveContainer" containerID="bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff"
Nov 03 17:02:52.678593 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.678572    5112 scope.go:122] "RemoveContainer" containerID="bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff"
Nov 03 17:02:52.678876 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:52.678854    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff\": container with ID starting with bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff not found: ID does not exist" containerID="bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff"
Nov 03 17:02:52.678965 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.678885    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff"} err="failed to get container status \"bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff\": rpc error: code = NotFound desc = could not find container \"bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff\": container with ID starting with bf79cd1e363ee2ba16760663eade4683688b88940d24e9990c1ebf673680f0ff not found: ID does not exist"
Nov 03 17:02:52.823506 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.823463    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"]
Nov 03 17:02:52.847155 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.847127    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c88d54-f979b8ff5-5cmjz"]
Nov 03 17:02:52.958579 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.958496    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"]
Nov 03 17:02:52.966083 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:52.966057    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c88d54-79448d5b7f-wnl9b"]
Nov 03 17:02:54.548592 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:54.548561    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6e4908-9236-4452-a3e1-3f26a5fc0449" path="/var/lib/kubelet/pods/2c6e4908-9236-4452-a3e1-3f26a5fc0449/volumes"
Nov 03 17:02:54.548989 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:54.548929    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5229c4c-b2fb-4418-ab4c-89458192001e" path="/var/lib/kubelet/pods/e5229c4c-b2fb-4418-ab4c-89458192001e/volumes"
Nov 03 17:02:54.812546 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:54.812456    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main" containerID="cri-o://aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb" gracePeriod=45
Nov 03 17:02:55.079435 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.079375    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 17:02:55.233341 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233312    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de833339-f19a-4c5c-8c68-78aa51789c40-kube-api-access-fg5g2\" (UniqueName: \"kubernetes.io/projected/de833339-f19a-4c5c-8c68-78aa51789c40-kube-api-access-fg5g2\") pod \"de833339-f19a-4c5c-8c68-78aa51789c40\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") "
Nov 03 17:02:55.233527 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233477    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-model-cache\") pod \"de833339-f19a-4c5c-8c68-78aa51789c40\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") "
Nov 03 17:02:55.233527 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233517    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-home\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-home\") pod \"de833339-f19a-4c5c-8c68-78aa51789c40\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") "
Nov 03 17:02:55.233662 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233584    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-tmp-dir\") pod \"de833339-f19a-4c5c-8c68-78aa51789c40\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") "
Nov 03 17:02:55.233662 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233630    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-dshm\") pod \"de833339-f19a-4c5c-8c68-78aa51789c40\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") "
Nov 03 17:02:55.233774 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233694    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de833339-f19a-4c5c-8c68-78aa51789c40-tls-certs\" (UniqueName: \"kubernetes.io/secret/de833339-f19a-4c5c-8c68-78aa51789c40-tls-certs\") pod \"de833339-f19a-4c5c-8c68-78aa51789c40\" (UID: \"de833339-f19a-4c5c-8c68-78aa51789c40\") "
Nov 03 17:02:55.233828 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233798    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-model-cache" pod "de833339-f19a-4c5c-8c68-78aa51789c40" (UID: "de833339-f19a-4c5c-8c68-78aa51789c40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:55.233885 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.233836    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-home" pod "de833339-f19a-4c5c-8c68-78aa51789c40" (UID: "de833339-f19a-4c5c-8c68-78aa51789c40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:55.234047 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.234022    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:55.234047 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.234042    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:55.234306 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.234277    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-tmp-dir" pod "de833339-f19a-4c5c-8c68-78aa51789c40" (UID: "de833339-f19a-4c5c-8c68-78aa51789c40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:55.235775 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.235751    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de833339-f19a-4c5c-8c68-78aa51789c40-kube-api-access-fg5g2" pod "de833339-f19a-4c5c-8c68-78aa51789c40" (UID: "de833339-f19a-4c5c-8c68-78aa51789c40"). InnerVolumeSpecName "kube-api-access-fg5g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:02:55.235924 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.235847    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de833339-f19a-4c5c-8c68-78aa51789c40-tls-certs" pod "de833339-f19a-4c5c-8c68-78aa51789c40" (UID: "de833339-f19a-4c5c-8c68-78aa51789c40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:02:55.236006 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.235952    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-dshm" pod "de833339-f19a-4c5c-8c68-78aa51789c40" (UID: "de833339-f19a-4c5c-8c68-78aa51789c40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:02:55.334815 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.334731    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:55.334815 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.334759    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de833339-f19a-4c5c-8c68-78aa51789c40-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:55.334815 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.334768    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de833339-f19a-4c5c-8c68-78aa51789c40-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:55.334815 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.334776    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg5g2\" (UniqueName: \"kubernetes.io/projected/de833339-f19a-4c5c-8c68-78aa51789c40-kube-api-access-fg5g2\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:02:55.680536 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.680496    5112 generic.go:356] "Generic (PLEG): container finished" podID="de833339-f19a-4c5c-8c68-78aa51789c40" containerID="aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb" exitCode=0
Nov 03 17:02:55.681015 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.680545    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"
Nov 03 17:02:55.681015 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.680566    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" event={"ID":"de833339-f19a-4c5c-8c68-78aa51789c40","Type":"ContainerDied","Data":"aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb"}
Nov 03 17:02:55.681015 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.680607    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl" event={"ID":"de833339-f19a-4c5c-8c68-78aa51789c40","Type":"ContainerDied","Data":"fc1bf1ec592b3496c80405a841c0364531780201d784d3927f5fbfd55e55254a"}
Nov 03 17:02:55.681015 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.680628    5112 scope.go:122] "RemoveContainer" containerID="aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb"
Nov 03 17:02:55.690634 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.690612    5112 scope.go:122] "RemoveContainer" containerID="aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb"
Nov 03 17:02:55.690907 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:02:55.690887    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb\": container with ID starting with aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb not found: ID does not exist" containerID="aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb"
Nov 03 17:02:55.690966 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.690917    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb"} err="failed to get container status \"aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb\": rpc error: code = NotFound desc = could not find container \"aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb\": container with ID starting with aa6f32104fcf22d7c3ccdb2a1d1b662bf7ced29ac6099d891842d11ac0c641cb not found: ID does not exist"
Nov 03 17:02:55.842612 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.842573    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"]
Nov 03 17:02:55.889570 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:55.889541    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c88d54/premium-w2-c88d54-kserve-56cb7cfbf4-894jl"]
Nov 03 17:02:56.548624 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:02:56.548561    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" path="/var/lib/kubelet/pods/de833339-f19a-4c5c-8c68-78aa51789c40/volumes"
Nov 03 17:03:07.883300 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:03:07.883255    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:03:08.548391 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.548362    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"]
Nov 03 17:03:08.548564 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.548540    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp" podUID="9cc4c08f-e98d-49ed-a69e-41fc144d0273" containerName="maas-api" containerID="cri-o://36e4c8b09da7d71e6b340ae9916eaaf814ad2b02f11d72073d5f93b52ab457c7" gracePeriod=30
Nov 03 17:03:08.762331 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.762294    5112 generic.go:356] "Generic (PLEG): container finished" podID="9cc4c08f-e98d-49ed-a69e-41fc144d0273" containerID="36e4c8b09da7d71e6b340ae9916eaaf814ad2b02f11d72073d5f93b52ab457c7" exitCode=0
Nov 03 17:03:08.762619 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.762590    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp" event={"ID":"9cc4c08f-e98d-49ed-a69e-41fc144d0273","Type":"ContainerDied","Data":"36e4c8b09da7d71e6b340ae9916eaaf814ad2b02f11d72073d5f93b52ab457c7"}
Nov 03 17:03:08.843998 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.843973    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 17:03:08.894904 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.894770    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-metrics-certs\") pod \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") "
Nov 03 17:03:08.896185 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.895533    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls\") pod \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") "
Nov 03 17:03:08.896185 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.895585    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc4c08f-e98d-49ed-a69e-41fc144d0273-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc4c08f-e98d-49ed-a69e-41fc144d0273-openshift-service-ca\") pod \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") "
Nov 03 17:03:08.896185 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.895621    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc4c08f-e98d-49ed-a69e-41fc144d0273-kube-api-access-2vlmq\" (UniqueName: \"kubernetes.io/projected/9cc4c08f-e98d-49ed-a69e-41fc144d0273-kube-api-access-2vlmq\") pod \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\" (UID: \"9cc4c08f-e98d-49ed-a69e-41fc144d0273\") "
Nov 03 17:03:08.896797 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.896768    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc4c08f-e98d-49ed-a69e-41fc144d0273-openshift-service-ca" pod "9cc4c08f-e98d-49ed-a69e-41fc144d0273" (UID: "9cc4c08f-e98d-49ed-a69e-41fc144d0273"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:03:08.899580 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.899549    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-metrics-certs" pod "9cc4c08f-e98d-49ed-a69e-41fc144d0273" (UID: "9cc4c08f-e98d-49ed-a69e-41fc144d0273"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:03:08.900355 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.900314    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls" pod "9cc4c08f-e98d-49ed-a69e-41fc144d0273" (UID: "9cc4c08f-e98d-49ed-a69e-41fc144d0273"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:03:08.901969 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.901944    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc4c08f-e98d-49ed-a69e-41fc144d0273-kube-api-access-2vlmq" pod "9cc4c08f-e98d-49ed-a69e-41fc144d0273" (UID: "9cc4c08f-e98d-49ed-a69e-41fc144d0273"). InnerVolumeSpecName "kube-api-access-2vlmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:03:08.997712 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.997661    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:08.997712 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.997705    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cc4c08f-e98d-49ed-a69e-41fc144d0273-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:08.997978 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.997724    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc4c08f-e98d-49ed-a69e-41fc144d0273-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:08.997978 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:08.997740    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vlmq\" (UniqueName: \"kubernetes.io/projected/9cc4c08f-e98d-49ed-a69e-41fc144d0273-kube-api-access-2vlmq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:09.768877 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:09.768837    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp" event={"ID":"9cc4c08f-e98d-49ed-a69e-41fc144d0273","Type":"ContainerDied","Data":"180d53b6d0abac4fd4f28490d36c04f751b5b1705ee1d50b250662765ab5ce1f"}
Nov 03 17:03:09.768877 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:09.768874    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"
Nov 03 17:03:09.769229 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:09.768907    5112 scope.go:122] "RemoveContainer" containerID="36e4c8b09da7d71e6b340ae9916eaaf814ad2b02f11d72073d5f93b52ab457c7"
Nov 03 17:03:09.933653 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:09.933612    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"]
Nov 03 17:03:09.980569 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:09.980534    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c88d54-5755fc8b5c-gshqp"]
Nov 03 17:03:10.549542 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:10.549497    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc4c08f-e98d-49ed-a69e-41fc144d0273" path="/var/lib/kubelet/pods/9cc4c08f-e98d-49ed-a69e-41fc144d0273/volumes"
Nov 03 17:03:21.690683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:21.690589    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"]
Nov 03 17:03:21.691145 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:21.690858    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" containerID="cri-o://aa6cd6a05882d68412c576b1f1c582fd156a9b89244fa262d7dcda877e5577fd" gracePeriod=30
Nov 03 17:03:25.193261 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:25.193224    5112 patch_prober.go:29] interesting pod/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 17:03:25.193772 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:25.193292    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 17:03:26.864806 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:26.864771    5112 generic.go:356] "Generic (PLEG): container finished" podID="ec52cc47-42d7-4e8f-b671-154086f71067" containerID="aa6cd6a05882d68412c576b1f1c582fd156a9b89244fa262d7dcda877e5577fd" exitCode=0
Nov 03 17:03:26.865234 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:26.864841    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" event={"ID":"ec52cc47-42d7-4e8f-b671-154086f71067","Type":"ContainerDied","Data":"aa6cd6a05882d68412c576b1f1c582fd156a9b89244fa262d7dcda877e5577fd"}
Nov 03 17:03:26.953178 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:26.953149    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 17:03:27.076183 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076103    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-istio-token\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-istio-token\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076183 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076139    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec52cc47-42d7-4e8f-b671-154086f71067-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec52cc47-42d7-4e8f-b671-154086f71067-istiod-ca-cert\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076183 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076168    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec52cc47-42d7-4e8f-b671-154086f71067-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec52cc47-42d7-4e8f-b671-154086f71067-istio-podinfo\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076227    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-certs\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076270    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-data\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076303    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-envoy\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076334    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-credential-socket\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076438    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-socket\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.076522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076491    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-kube-api-access-bxhcz\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-kube-api-access-bxhcz\") pod \"ec52cc47-42d7-4e8f-b671-154086f71067\" (UID: \"ec52cc47-42d7-4e8f-b671-154086f71067\") "
Nov 03 17:03:27.077046 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076567    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-certs" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:03:27.077046 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076660    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-data" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:03:27.077046 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076703    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec52cc47-42d7-4e8f-b671-154086f71067-istiod-ca-cert" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:03:27.077046 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.076842    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-credential-socket" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:03:27.077298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.077057    5112 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec52cc47-42d7-4e8f-b671-154086f71067-istiod-ca-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.077298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.077078    5112 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.077298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.077092    5112 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-data\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.077298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.077108    5112 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-credential-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.077521 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.077444    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-socket" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:03:27.078649 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.078624    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-istio-token" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:03:27.078767 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.078650    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec52cc47-42d7-4e8f-b671-154086f71067-istio-podinfo" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:03:27.078816 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.078784    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-kube-api-access-bxhcz" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "kube-api-access-bxhcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:03:27.079404 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.079389    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-envoy" pod "ec52cc47-42d7-4e8f-b671-154086f71067" (UID: "ec52cc47-42d7-4e8f-b671-154086f71067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:03:27.178102 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.178065    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhcz\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-kube-api-access-bxhcz\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.178102 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.178097    5112 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec52cc47-42d7-4e8f-b671-154086f71067-istio-token\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.178102 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.178109    5112 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec52cc47-42d7-4e8f-b671-154086f71067-istio-podinfo\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.178360 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.178119    5112 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-istio-envoy\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.178360 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.178127    5112 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec52cc47-42d7-4e8f-b671-154086f71067-workload-socket\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:27.872569 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.872537    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"
Nov 03 17:03:27.873087 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.872577    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l" event={"ID":"ec52cc47-42d7-4e8f-b671-154086f71067","Type":"ContainerDied","Data":"561681769af1b865a93847be0ce59d7c1fdce2a7da8a74f37f08f87d2845422d"}
Nov 03 17:03:27.873087 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:27.872665    5112 scope.go:122] "RemoveContainer" containerID="aa6cd6a05882d68412c576b1f1c582fd156a9b89244fa262d7dcda877e5577fd"
Nov 03 17:03:28.081599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:28.081555    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"]
Nov 03 17:03:28.128562 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:28.128488    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c88d54-openshift-default-6d8d987d6f-f264l"]
Nov 03 17:03:28.548599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:28.548566    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" path="/var/lib/kubelet/pods/ec52cc47-42d7-4e8f-b671-154086f71067/volumes"
Nov 03 17:03:29.781538 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781499    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b46f6b789-485nz"]
Nov 03 17:03:29.781948 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781935    5112 cpu_manager.go:414] "containerMap: removing container" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main"
Nov 03 17:03:29.781992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781948    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main"
Nov 03 17:03:29.781992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781963    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b0ff4aac-032d-483d-8ea3-648619a23e2d" containerName="payload-processing"
Nov 03 17:03:29.781992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781968    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ff4aac-032d-483d-8ea3-648619a23e2d" containerName="payload-processing"
Nov 03 17:03:29.781992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781976    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2478e2ce-c49e-483d-9ce4-60f0283a323e" containerName="maas-api"
Nov 03 17:03:29.781992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781981    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2478e2ce-c49e-483d-9ce4-60f0283a323e" containerName="maas-api"
Nov 03 17:03:29.781992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781989    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2c6e4908-9236-4452-a3e1-3f26a5fc0449" containerName="payload-processing"
Nov 03 17:03:29.781992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.781994    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6e4908-9236-4452-a3e1-3f26a5fc0449" containerName="payload-processing"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782001    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782006    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782011    5112 cpu_manager.go:414] "containerMap: removing container" podUID="e5229c4c-b2fb-4418-ab4c-89458192001e" containerName="payload-pre-processing"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782016    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5229c4c-b2fb-4418-ab4c-89458192001e" containerName="payload-pre-processing"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782024    5112 cpu_manager.go:414] "containerMap: removing container" podUID="8648b34b-c21b-49e6-b1d9-4b12b8df2c24" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782030    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8648b34b-c21b-49e6-b1d9-4b12b8df2c24" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782038    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782045    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782050    5112 cpu_manager.go:414] "containerMap: removing container" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782054    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782066    5112 cpu_manager.go:414] "containerMap: removing container" podUID="6ea32444-b868-4322-81b3-4fd19c09387b" containerName="main"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782070    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea32444-b868-4322-81b3-4fd19c09387b" containerName="main"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782079    5112 cpu_manager.go:414] "containerMap: removing container" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782083    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782090    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782096    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782103    5112 cpu_manager.go:414] "containerMap: removing container" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782108    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782117    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782122    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782128    5112 cpu_manager.go:414] "containerMap: removing container" podUID="9cc4c08f-e98d-49ed-a69e-41fc144d0273" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782133    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc4c08f-e98d-49ed-a69e-41fc144d0273" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782139    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782143    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782149    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782154    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782163    5112 cpu_manager.go:414] "containerMap: removing container" podUID="acc9d684-66ba-485c-a926-deaa20d41fd6" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782167    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc9d684-66ba-485c-a926-deaa20d41fd6" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782173    5112 cpu_manager.go:414] "containerMap: removing container" podUID="8d628a4f-6ead-4e79-b7cd-57a2806f2e21" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782177    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d628a4f-6ead-4e79-b7cd-57a2806f2e21" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782184    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2075f8cb-c6c3-4c64-ac44-f153d43aeb70" containerName="payload-pre-processing"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782189    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2075f8cb-c6c3-4c64-ac44-f153d43aeb70" containerName="payload-pre-processing"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782212    5112 cpu_manager.go:414] "containerMap: removing container" podUID="105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" containerName="payload-pre-processing"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782217    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" containerName="payload-pre-processing"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782224    5112 cpu_manager.go:414] "containerMap: removing container" podUID="8052eca5-aedb-4c3d-b37f-051c24042490" containerName="maas-api"
Nov 03 17:03:29.782228 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782228    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8052eca5-aedb-4c3d-b37f-051c24042490" containerName="maas-api"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782298    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d861a3-b90a-456f-9f54-ff31f6c09f1e" containerName="istio-proxy"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782307    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc4c08f-e98d-49ed-a69e-41fc144d0273" containerName="maas-api"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782314    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782319    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782324    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782329    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc9d684-66ba-485c-a926-deaa20d41fd6" containerName="maas-api"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782337    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d628a4f-6ead-4e79-b7cd-57a2806f2e21" containerName="maas-api"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782343    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2075f8cb-c6c3-4c64-ac44-f153d43aeb70" containerName="payload-pre-processing"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782348    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="105b3420-4eb9-4bb0-bdc3-afb6bee8f9eb" containerName="payload-pre-processing"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782353    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="8052eca5-aedb-4c3d-b37f-051c24042490" containerName="maas-api"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782359    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="de833339-f19a-4c5c-8c68-78aa51789c40" containerName="main"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782365    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ff4aac-032d-483d-8ea3-648619a23e2d" containerName="payload-processing"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782369    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2478e2ce-c49e-483d-9ce4-60f0283a323e" containerName="maas-api"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782374    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6e4908-9236-4452-a3e1-3f26a5fc0449" containerName="payload-processing"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782379    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5229c4c-b2fb-4418-ab4c-89458192001e" containerName="payload-pre-processing"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782385    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="8648b34b-c21b-49e6-b1d9-4b12b8df2c24" containerName="maas-api"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782390    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782395    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec52cc47-42d7-4e8f-b671-154086f71067" containerName="istio-proxy"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782400    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea32444-b868-4322-81b3-4fd19c09387b" containerName="main"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782405    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="99609b93-a9c5-4641-a96d-2b7e7aa6baa1" containerName="main"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782410    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7af5ace-a5cd-4b3b-b88b-61969a16ea82" containerName="main"
Nov 03 17:03:29.783397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.782415    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fbeb18-4414-43b8-a190-36beb56e4193" containerName="istio-proxy"
Nov 03 17:03:29.787192 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.787172    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:29.885601 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.885569    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 17:03:29.885601 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.885584    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 17:03:29.885842 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.885662    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rfldq\""
Nov 03 17:03:29.885842 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.885773    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 17:03:29.888182 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.888162    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-485nz"]
Nov 03 17:03:29.908798 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.908767    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-webhook-cert\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:29.908921 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.908901    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-metrics-certs\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:29.908997 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:29.908980    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jfg\" (UniqueName: \"kubernetes.io/projected/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-kube-api-access-t4jfg\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.010511 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.010476    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-metrics-certs\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.010721 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.010545    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jfg\" (UniqueName: \"kubernetes.io/projected/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-kube-api-access-t4jfg\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.010721 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.010576    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-webhook-cert\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.013101 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.013073    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-webhook-cert\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.013266 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.013244    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-metrics-certs\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.067977 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.067901    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jfg\" (UniqueName: \"kubernetes.io/projected/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-kube-api-access-t4jfg\") pod \"maas-controller-6b46f6b789-485nz\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") " pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.100101 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.100070    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:30.473254 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.473223    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-485nz"]
Nov 03 17:03:30.502838 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:03:30.502806    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5b6a8b_5b75_4f66_8b2d_cff711290ef7.slice/crio-d9acc4d49919824ac1720230dca290c8e3e3374123365832bdfcf5061f5a651c WatchSource:0}: Error finding container d9acc4d49919824ac1720230dca290c8e3e3374123365832bdfcf5061f5a651c: Status 404 returned error can't find the container with id d9acc4d49919824ac1720230dca290c8e3e3374123365832bdfcf5061f5a651c
Nov 03 17:03:30.504157 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.504139    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:03:30.892635 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:30.892542    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-485nz" event={"ID":"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7","Type":"ContainerStarted","Data":"d9acc4d49919824ac1720230dca290c8e3e3374123365832bdfcf5061f5a651c"}
Nov 03 17:03:33.912777 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:33.912739    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-485nz" event={"ID":"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7","Type":"ContainerStarted","Data":"5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35"}
Nov 03 17:03:33.913193 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:33.912871    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:34.035186 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:34.035129    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b46f6b789-485nz" podStartSLOduration=2.32778262 podStartE2EDuration="5.035115535s" totalImagesPullingTime="2.707332915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:34.033512791 +0000 UTC m=+3175.612816662" watchObservedRunningTime="2026-11-03 17:03:34.035115535 +0000 UTC m=+3175.614419390"
Nov 03 17:03:37.883991 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:03:37.883954    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:03:44.919791 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:44.919757    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:03:50.647558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.647514    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"]
Nov 03 17:03:50.648244 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.648219    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:50.648244 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.648244    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:50.648410 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.648283    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:50.648410 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.648291    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:50.648516 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.648414    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2917f91-5099-4d4b-9e7e-2d7c95ebb38f" containerName="cleanup"
Nov 03 17:03:50.648516 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.648431    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67ba266-b9d5-4bf6-9444-a858edd52bb0" containerName="cleanup"
Nov 03 17:03:50.653172 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.653151    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.702862 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.702838    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.702990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.702880    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.702990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.702897    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.702990 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.702968    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c132a838-d7ba-4445-a3e7-f5e4955d4688-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.703138 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.703000    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.703138 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.703023    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.703138 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.703117    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvw9x\" (UniqueName: \"kubernetes.io/projected/c132a838-d7ba-4445-a3e7-f5e4955d4688-kube-api-access-nvw9x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.703278 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.703157    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.703278 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.703237    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.704086 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.704070    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mw47n\""
Nov 03 17:03:50.755110 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.755082    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"]
Nov 03 17:03:50.804130 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804099    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804130 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804131    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804151    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c132a838-d7ba-4445-a3e7-f5e4955d4688-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804166    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804184    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804238    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvw9x\" (UniqueName: \"kubernetes.io/projected/c132a838-d7ba-4445-a3e7-f5e4955d4688-kube-api-access-nvw9x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804267    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804303    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804676 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804390    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804676 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804604    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804676 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804662    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804829    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.804953 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804927    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c132a838-d7ba-4445-a3e7-f5e4955d4688-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.805015 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.804959    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.806727 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.806703    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.806795 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.806780    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.861085 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.861056    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c132a838-d7ba-4445-a3e7-f5e4955d4688-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.861249 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.861159    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvw9x\" (UniqueName: \"kubernetes.io/projected/c132a838-d7ba-4445-a3e7-f5e4955d4688-kube-api-access-nvw9x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2jzfr\" (UID: \"c132a838-d7ba-4445-a3e7-f5e4955d4688\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:50.966316 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:50.966233    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:51.332342 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.332263    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"]
Nov 03 17:03:51.336983 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.336965    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.392136 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.392100    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"]
Nov 03 17:03:51.392360 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.392301    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:03:51.392438 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.392380    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:03:51.394438 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.394410    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"]
Nov 03 17:03:51.409927 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.409895    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcg8k\" (UniqueName: \"kubernetes.io/projected/014a8a9d-5a65-4f79-8303-a24f771c97ea-kube-api-access-pcg8k\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.410060 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.409940    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.410060 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.409968    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.440430 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:03:51.440396    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc132a838_d7ba_4445_a3e7_f5e4955d4688.slice/crio-70dbb0d953277a8e725e84cd0fb98d4f99ac0660b1d11ed1b9aa82e1e6424d7e WatchSource:0}: Error finding container 70dbb0d953277a8e725e84cd0fb98d4f99ac0660b1d11ed1b9aa82e1e6424d7e: Status 404 returned error can't find the container with id 70dbb0d953277a8e725e84cd0fb98d4f99ac0660b1d11ed1b9aa82e1e6424d7e
Nov 03 17:03:51.443045 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.443010    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:03:51.443126 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.443079    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:03:51.443126 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.443108    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:03:51.511366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.511337    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.511523 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.511393    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.511589 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.511524    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcg8k\" (UniqueName: \"kubernetes.io/projected/014a8a9d-5a65-4f79-8303-a24f771c97ea-kube-api-access-pcg8k\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.512079 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.512057    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.513667 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.513645    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.569131 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.569099    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcg8k\" (UniqueName: \"kubernetes.io/projected/014a8a9d-5a65-4f79-8303-a24f771c97ea-kube-api-access-pcg8k\") pod \"maas-api-revoke-sub-85237e45-7247h\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:51.650235 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:51.650180    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:52.012971 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:52.012847    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" event={"ID":"c132a838-d7ba-4445-a3e7-f5e4955d4688","Type":"ContainerStarted","Data":"c088fba297d946875674da58d9e6753d7de7369404a0667c4b88d04e64394499"}
Nov 03 17:03:52.012971 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:52.012922    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" event={"ID":"c132a838-d7ba-4445-a3e7-f5e4955d4688","Type":"ContainerStarted","Data":"70dbb0d953277a8e725e84cd0fb98d4f99ac0660b1d11ed1b9aa82e1e6424d7e"}
Nov 03 17:03:52.029475 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:52.029447    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"]
Nov 03 17:03:52.031510 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:03:52.031473    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014a8a9d_5a65_4f79_8303_a24f771c97ea.slice/crio-2da227d7d3f6939b032b00bce9cc959a40a53370aef35a2afb383176dba0066b WatchSource:0}: Error finding container 2da227d7d3f6939b032b00bce9cc959a40a53370aef35a2afb383176dba0066b: Status 404 returned error can't find the container with id 2da227d7d3f6939b032b00bce9cc959a40a53370aef35a2afb383176dba0066b
Nov 03 17:03:52.151965 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:52.151909    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podStartSLOduration=2.151896004 podStartE2EDuration="2.151896004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:52.151424163 +0000 UTC m=+3193.730728029" watchObservedRunningTime="2026-11-03 17:03:52.151896004 +0000 UTC m=+3193.731199859"
Nov 03 17:03:52.976223 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:52.966451    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:03:52.976223 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:52.968721    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 17:03:52.976223 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:52.968792    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 17:03:53.021786 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:53.021738    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h" event={"ID":"014a8a9d-5a65-4f79-8303-a24f771c97ea","Type":"ContainerStarted","Data":"f39b38f485c4aef09489be0bfab5bd5d56e5b293f3be79a21ed2cb510b09cb48"}
Nov 03 17:03:53.021963 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:53.021814    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h" event={"ID":"014a8a9d-5a65-4f79-8303-a24f771c97ea","Type":"ContainerStarted","Data":"2da227d7d3f6939b032b00bce9cc959a40a53370aef35a2afb383176dba0066b"}
Nov 03 17:03:53.143308 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:53.143257    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h" podStartSLOduration=2.143244131 podStartE2EDuration="2.143244131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:53.142432269 +0000 UTC m=+3194.721736149" watchObservedRunningTime="2026-11-03 17:03:53.143244131 +0000 UTC m=+3194.722547986"
Nov 03 17:03:53.979234 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:53.972662    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 17:03:53.979234 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:53.972732    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 17:03:54.029105 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:54.029069    5112 generic.go:356] "Generic (PLEG): container finished" podID="014a8a9d-5a65-4f79-8303-a24f771c97ea" containerID="f39b38f485c4aef09489be0bfab5bd5d56e5b293f3be79a21ed2cb510b09cb48" exitCode=0
Nov 03 17:03:54.029272 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:54.029158    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h" event={"ID":"014a8a9d-5a65-4f79-8303-a24f771c97ea","Type":"ContainerDied","Data":"f39b38f485c4aef09489be0bfab5bd5d56e5b293f3be79a21ed2cb510b09cb48"}
Nov 03 17:03:54.975224 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:54.970631    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 17:03:54.975224 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:54.970697    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 17:03:55.253935 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.253909    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:55.351102 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.351063    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014a8a9d-5a65-4f79-8303-a24f771c97ea-kube-api-access-pcg8k\" (UniqueName: \"kubernetes.io/projected/014a8a9d-5a65-4f79-8303-a24f771c97ea-kube-api-access-pcg8k\") pod \"014a8a9d-5a65-4f79-8303-a24f771c97ea\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") "
Nov 03 17:03:55.351102 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.351104    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-xks-ca\") pod \"014a8a9d-5a65-4f79-8303-a24f771c97ea\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") "
Nov 03 17:03:55.351350 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.351162    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-openshift-ca\") pod \"014a8a9d-5a65-4f79-8303-a24f771c97ea\" (UID: \"014a8a9d-5a65-4f79-8303-a24f771c97ea\") "
Nov 03 17:03:55.351570 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.351541    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-openshift-ca" pod "014a8a9d-5a65-4f79-8303-a24f771c97ea" (UID: "014a8a9d-5a65-4f79-8303-a24f771c97ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:03:55.353324 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.353295    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014a8a9d-5a65-4f79-8303-a24f771c97ea-kube-api-access-pcg8k" pod "014a8a9d-5a65-4f79-8303-a24f771c97ea" (UID: "014a8a9d-5a65-4f79-8303-a24f771c97ea"). InnerVolumeSpecName "kube-api-access-pcg8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:03:55.353419 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.353301    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-xks-ca" pod "014a8a9d-5a65-4f79-8303-a24f771c97ea" (UID: "014a8a9d-5a65-4f79-8303-a24f771c97ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:03:55.452705 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.452672    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcg8k\" (UniqueName: \"kubernetes.io/projected/014a8a9d-5a65-4f79-8303-a24f771c97ea-kube-api-access-pcg8k\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:55.452705 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.452700    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:55.452705 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.452710    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014a8a9d-5a65-4f79-8303-a24f771c97ea-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:03:55.979675 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.979620    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 17:03:55.979900 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:55.979703    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 17:03:56.048035 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:56.047227    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h" event={"ID":"014a8a9d-5a65-4f79-8303-a24f771c97ea","Type":"ContainerDied","Data":"2da227d7d3f6939b032b00bce9cc959a40a53370aef35a2afb383176dba0066b"}
Nov 03 17:03:56.048035 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:56.047290    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da227d7d3f6939b032b00bce9cc959a40a53370aef35a2afb383176dba0066b"
Nov 03 17:03:56.048035 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:56.047358    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"
Nov 03 17:03:56.967994 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:56.967547    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 17:03:56.967994 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:56.967609    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 17:03:57.967410 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:57.967368    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 17:03:57.967600 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:57.967441    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 17:03:59.967599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:59.967554    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:03:59.968137 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:03:59.967628    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:04:01.010371 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:01.010323    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:04:01.010912 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:01.010403    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:04:01.805390 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:01.805356    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:04:01.805812 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:01.805782    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:04:01.807064 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:01.807046    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr"
Nov 03 17:04:03.663094 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:03.663054    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-485nz"]
Nov 03 17:04:03.663606 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:03.663315    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6b46f6b789-485nz" podUID="9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" containerName="manager" containerID="cri-o://5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35" gracePeriod=30
Nov 03 17:04:03.940810 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:03.940785    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:04:04.062954 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.062922    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-kube-api-access-t4jfg\" (UniqueName: \"kubernetes.io/projected/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-kube-api-access-t4jfg\") pod \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") "
Nov 03 17:04:04.063155 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.062988    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-webhook-cert\") pod \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") "
Nov 03 17:04:04.063155 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.063118    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-metrics-certs\") pod \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\" (UID: \"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7\") "
Nov 03 17:04:04.065790 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.065668    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-metrics-certs" pod "9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" (UID: "9a5b6a8b-5b75-4f66-8b2d-cff711290ef7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:04:04.065790 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.065739    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-webhook-cert" pod "9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" (UID: "9a5b6a8b-5b75-4f66-8b2d-cff711290ef7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:04:04.065790 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.065763    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-kube-api-access-t4jfg" pod "9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" (UID: "9a5b6a8b-5b75-4f66-8b2d-cff711290ef7"). InnerVolumeSpecName "kube-api-access-t4jfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:04:04.095467 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.095436    5112 generic.go:356] "Generic (PLEG): container finished" podID="9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" containerID="5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35" exitCode=0
Nov 03 17:04:04.095673 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.095482    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-485nz"
Nov 03 17:04:04.095673 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.095512    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-485nz" event={"ID":"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7","Type":"ContainerDied","Data":"5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35"}
Nov 03 17:04:04.095673 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.095570    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-485nz" event={"ID":"9a5b6a8b-5b75-4f66-8b2d-cff711290ef7","Type":"ContainerDied","Data":"d9acc4d49919824ac1720230dca290c8e3e3374123365832bdfcf5061f5a651c"}
Nov 03 17:04:04.095673 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.095598    5112 scope.go:122] "RemoveContainer" containerID="5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35"
Nov 03 17:04:04.106879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.106847    5112 scope.go:122] "RemoveContainer" containerID="5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35"
Nov 03 17:04:04.107180 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:04:04.107158    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35\": container with ID starting with 5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35 not found: ID does not exist" containerID="5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35"
Nov 03 17:04:04.107316 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.107187    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35"} err="failed to get container status \"5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35\": rpc error: code = NotFound desc = could not find container \"5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35\": container with ID starting with 5917af2828959c0fcaf63f889bc851cf52de243d297a59214076cfe1dfb31d35 not found: ID does not exist"
Nov 03 17:04:04.164292 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.164262    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jfg\" (UniqueName: \"kubernetes.io/projected/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-kube-api-access-t4jfg\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:04:04.164292 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.164289    5112 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-webhook-cert\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:04:04.164292 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.164298    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:04:04.255705 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.255621    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-485nz"]
Nov 03 17:04:04.261000 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.260971    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-485nz"]
Nov 03 17:04:04.555260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:04.549712    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" path="/var/lib/kubelet/pods/9a5b6a8b-5b75-4f66-8b2d-cff711290ef7/volumes"
Nov 03 17:04:05.608542 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.608498    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b46f6b789-vpnvw"]
Nov 03 17:04:05.609178 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.609154    5112 cpu_manager.go:414] "containerMap: removing container" podUID="9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" containerName="manager"
Nov 03 17:04:05.609317 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.609180    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" containerName="manager"
Nov 03 17:04:05.609317 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.609223    5112 cpu_manager.go:414] "containerMap: removing container" podUID="014a8a9d-5a65-4f79-8303-a24f771c97ea" containerName="revoke-keys"
Nov 03 17:04:05.609317 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.609232    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014a8a9d-5a65-4f79-8303-a24f771c97ea" containerName="revoke-keys"
Nov 03 17:04:05.609469 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.609378    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5b6a8b-5b75-4f66-8b2d-cff711290ef7" containerName="manager"
Nov 03 17:04:05.609469 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.609396    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="014a8a9d-5a65-4f79-8303-a24f771c97ea" containerName="revoke-keys"
Nov 03 17:04:05.615593 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.615571    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.713268 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.713236    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 17:04:05.713461 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.713351    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 17:04:05.713461 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.713398    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 17:04:05.713461 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.713434    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rfldq\""
Nov 03 17:04:05.718081 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.718053    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-vpnvw"]
Nov 03 17:04:05.780049 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.780020    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75918970-083d-41a0-aa18-05a395d1f934-metrics-certs\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.780049 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.780052    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75918970-083d-41a0-aa18-05a395d1f934-webhook-cert\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.780279 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.780077    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7llc\" (UniqueName: \"kubernetes.io/projected/75918970-083d-41a0-aa18-05a395d1f934-kube-api-access-m7llc\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.881164 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.881069    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7llc\" (UniqueName: \"kubernetes.io/projected/75918970-083d-41a0-aa18-05a395d1f934-kube-api-access-m7llc\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.881377 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.881266    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75918970-083d-41a0-aa18-05a395d1f934-metrics-certs\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.881377 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.881297    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75918970-083d-41a0-aa18-05a395d1f934-webhook-cert\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.883786 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.883763    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75918970-083d-41a0-aa18-05a395d1f934-metrics-certs\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.883894 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.883807    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75918970-083d-41a0-aa18-05a395d1f934-webhook-cert\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:05.937260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:05.937220    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7llc\" (UniqueName: \"kubernetes.io/projected/75918970-083d-41a0-aa18-05a395d1f934-kube-api-access-m7llc\") pod \"maas-controller-6b46f6b789-vpnvw\" (UID: \"75918970-083d-41a0-aa18-05a395d1f934\") " pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:06.230028 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:06.229992    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:06.603950 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:06.603926    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b46f6b789-vpnvw"]
Nov 03 17:04:06.605862 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:04:06.605829    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75918970_083d_41a0_aa18_05a395d1f934.slice/crio-00ce070ed1fc79942aa58870f729d6f7bae14b916695514dda8095e82713d611 WatchSource:0}: Error finding container 00ce070ed1fc79942aa58870f729d6f7bae14b916695514dda8095e82713d611: Status 404 returned error can't find the container with id 00ce070ed1fc79942aa58870f729d6f7bae14b916695514dda8095e82713d611
Nov 03 17:04:07.114879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:07.114841    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-vpnvw" event={"ID":"75918970-083d-41a0-aa18-05a395d1f934","Type":"ContainerStarted","Data":"00ce070ed1fc79942aa58870f729d6f7bae14b916695514dda8095e82713d611"}
Nov 03 17:04:07.882878 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:04:07.882794    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:04:08.121584 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:08.121549    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b46f6b789-vpnvw" event={"ID":"75918970-083d-41a0-aa18-05a395d1f934","Type":"ContainerStarted","Data":"bb6ba885edfaf1383f3169ba364770ffbeaefd41ad244fbc1542bf93c7d2cbaf"}
Nov 03 17:04:08.123946 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:08.121617    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:08.238184 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:08.238126    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b46f6b789-vpnvw" podStartSLOduration=2.32986278 podStartE2EDuration="3.238110818s" totalImagesPullingTime="908.248038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:04:08.236160889 +0000 UTC m=+3209.815464764" watchObservedRunningTime="2026-11-03 17:04:08.238110818 +0000 UTC m=+3209.817414719"
Nov 03 17:04:19.130820 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:04:19.130789    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b46f6b789-vpnvw"
Nov 03 17:04:37.883247 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:04:37.883206    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:05:07.407145 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.407058    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"]
Nov 03 17:05:07.412184 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.412163    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.466947 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.466912    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"]
Nov 03 17:05:07.511513 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.511475    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:05:07.511513 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.511495    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:05:07.568879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.568844    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.569054 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.568891    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.569104 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.569083    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrvwn\" (UniqueName: \"kubernetes.io/projected/61d3a30d-7196-4142-8f4a-90ac9273ac2e-kube-api-access-nrvwn\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.670270 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.670169    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvwn\" (UniqueName: \"kubernetes.io/projected/61d3a30d-7196-4142-8f4a-90ac9273ac2e-kube-api-access-nrvwn\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.670442 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.670285    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.670442 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.670338    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.671035 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.671008    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.672710 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.672689    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.727043 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:07.727008    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrvwn\" (UniqueName: \"kubernetes.io/projected/61d3a30d-7196-4142-8f4a-90ac9273ac2e-kube-api-access-nrvwn\") pod \"maas-api-revoke-sub-c1e271a9-w9t7l\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:07.882218 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:05:07.882167    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:05:08.025272 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:08.025170    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:08.394918 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:08.394893    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"]
Nov 03 17:05:08.396844 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:05:08.396810    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d3a30d_7196_4142_8f4a_90ac9273ac2e.slice/crio-ab13527c3612e1a7749aded83643dd45eb8237775375b4220bf872b818ad3afb WatchSource:0}: Error finding container ab13527c3612e1a7749aded83643dd45eb8237775375b4220bf872b818ad3afb: Status 404 returned error can't find the container with id ab13527c3612e1a7749aded83643dd45eb8237775375b4220bf872b818ad3afb
Nov 03 17:05:08.456901 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:08.456867    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l" event={"ID":"61d3a30d-7196-4142-8f4a-90ac9273ac2e","Type":"ContainerStarted","Data":"ab13527c3612e1a7749aded83643dd45eb8237775375b4220bf872b818ad3afb"}
Nov 03 17:05:09.464558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:09.464465    5112 generic.go:356] "Generic (PLEG): container finished" podID="61d3a30d-7196-4142-8f4a-90ac9273ac2e" containerID="dfea03295a5575e54ccda936ecd70c718a9aa2eb7b8fe8d53fd8db52c3967826" exitCode=0
Nov 03 17:05:09.465015 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:09.464556    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l" event={"ID":"61d3a30d-7196-4142-8f4a-90ac9273ac2e","Type":"ContainerDied","Data":"dfea03295a5575e54ccda936ecd70c718a9aa2eb7b8fe8d53fd8db52c3967826"}
Nov 03 17:05:10.620047 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.620021    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:10.802091 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.801978    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-openshift-ca\") pod \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") "
Nov 03 17:05:10.802320 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.802104    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-xks-ca\") pod \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") "
Nov 03 17:05:10.802320 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.802166    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d3a30d-7196-4142-8f4a-90ac9273ac2e-kube-api-access-nrvwn\" (UniqueName: \"kubernetes.io/projected/61d3a30d-7196-4142-8f4a-90ac9273ac2e-kube-api-access-nrvwn\") pod \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\" (UID: \"61d3a30d-7196-4142-8f4a-90ac9273ac2e\") "
Nov 03 17:05:10.802513 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.802483    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-openshift-ca" pod "61d3a30d-7196-4142-8f4a-90ac9273ac2e" (UID: "61d3a30d-7196-4142-8f4a-90ac9273ac2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:05:10.802617 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.802602    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:10.804371 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.804345    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-xks-ca" pod "61d3a30d-7196-4142-8f4a-90ac9273ac2e" (UID: "61d3a30d-7196-4142-8f4a-90ac9273ac2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:05:10.804371 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.804358    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d3a30d-7196-4142-8f4a-90ac9273ac2e-kube-api-access-nrvwn" pod "61d3a30d-7196-4142-8f4a-90ac9273ac2e" (UID: "61d3a30d-7196-4142-8f4a-90ac9273ac2e"). InnerVolumeSpecName "kube-api-access-nrvwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:05:10.903804 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.903756    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d3a30d-7196-4142-8f4a-90ac9273ac2e-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:10.903804 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:10.903800    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrvwn\" (UniqueName: \"kubernetes.io/projected/61d3a30d-7196-4142-8f4a-90ac9273ac2e-kube-api-access-nrvwn\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:11.476792 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:11.476759    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"
Nov 03 17:05:11.476992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:11.476794    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l" event={"ID":"61d3a30d-7196-4142-8f4a-90ac9273ac2e","Type":"ContainerDied","Data":"ab13527c3612e1a7749aded83643dd45eb8237775375b4220bf872b818ad3afb"}
Nov 03 17:05:11.476992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:11.476879    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab13527c3612e1a7749aded83643dd45eb8237775375b4220bf872b818ad3afb"
Nov 03 17:05:21.383731 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.383697    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"]
Nov 03 17:05:21.384267 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.384146    5112 cpu_manager.go:414] "containerMap: removing container" podUID="61d3a30d-7196-4142-8f4a-90ac9273ac2e" containerName="revoke-keys"
Nov 03 17:05:21.384267 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.384158    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d3a30d-7196-4142-8f4a-90ac9273ac2e" containerName="revoke-keys"
Nov 03 17:05:21.384350 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.384285    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d3a30d-7196-4142-8f4a-90ac9273ac2e" containerName="revoke-keys"
Nov 03 17:05:21.387130 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.387104    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504107 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504074    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504331 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504126    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504331 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504256    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504451 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504393    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6h9\" (UniqueName: \"kubernetes.io/projected/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-kube-api-access-pr6h9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504513 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504462    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504513 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504503    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504585 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504566    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504623 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504606    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.504673 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.504656    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.588189 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.588150    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"]
Nov 03 17:05:21.605392 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605359    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6h9\" (UniqueName: \"kubernetes.io/projected/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-kube-api-access-pr6h9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605400    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605423    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605441    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605462    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605500    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605522    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605558 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605559    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605784 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605587    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.605936 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.605908    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.606038 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.606018    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.606274 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.606252    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.606448 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.606310    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.606620 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.606598    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.608303 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.608281    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.608449 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.608428    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.662569 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.662527    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.662746 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.662724    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6h9\" (UniqueName: \"kubernetes.io/projected/106b94ab-7184-4c6b-b0e4-3bf6a4b763a0-kube-api-access-pr6h9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-2wnwx\" (UID: \"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:21.699971 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:21.699936    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:22.126939 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.126913    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"]
Nov 03 17:05:22.128898 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:05:22.128869    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106b94ab_7184_4c6b_b0e4_3bf6a4b763a0.slice/crio-b7caa59fda70f300b78dd46be30f86101a48cc01079d3fe0d12c07d10bdeb6af WatchSource:0}: Error finding container b7caa59fda70f300b78dd46be30f86101a48cc01079d3fe0d12c07d10bdeb6af: Status 404 returned error can't find the container with id b7caa59fda70f300b78dd46be30f86101a48cc01079d3fe0d12c07d10bdeb6af
Nov 03 17:05:22.131073 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.131036    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:05:22.131133 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.131103    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:05:22.131168 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.131131    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:05:22.540004 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.539957    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" event={"ID":"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0","Type":"ContainerStarted","Data":"fd8f4007cfc0fba3ece07a0a3f78bf1526b87b2e7d2767b32a67200d81db1a48"}
Nov 03 17:05:22.540610 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.540059    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" event={"ID":"106b94ab-7184-4c6b-b0e4-3bf6a4b763a0","Type":"ContainerStarted","Data":"b7caa59fda70f300b78dd46be30f86101a48cc01079d3fe0d12c07d10bdeb6af"}
Nov 03 17:05:22.700865 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.700830    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:22.757902 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:22.757826    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podStartSLOduration=1.757793338 podStartE2EDuration="1.757793338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:22.708820509 +0000 UTC m=+3284.288124365" watchObservedRunningTime="2026-11-03 17:05:22.757793338 +0000 UTC m=+3284.337097195"
Nov 03 17:05:23.702340 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:23.702299    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:05:23.702769 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:23.702384    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 17:05:24.700607 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:24.700568    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:05:24.700794 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:24.700640    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 17:05:25.700675 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:25.700632    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:05:25.701090 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:25.700699    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 17:05:26.701023 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:26.700985    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:05:26.701443 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:26.701039    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 17:05:27.700900 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:27.700846    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:05:27.701121 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:27.700922    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 17:05:28.700796 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:28.700759    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 17:05:28.701011 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:28.700820    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 17:05:30.394320 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:30.394283    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:30.394834 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:30.394620    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:30.395728 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:30.395699    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx"
Nov 03 17:05:37.883359 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:05:37.883321    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:05:39.360059 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:39.360031    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:05:39.370997 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:39.370968    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:05:39.493698 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:39.493667    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:05:39.505816 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:39.505784    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:05:42.480248 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.480189    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"]
Nov 03 17:05:42.484137 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.484119    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.539179 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.539143    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"]
Nov 03 17:05:42.582875 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.582841    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:05:42.583042 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.583012    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:05:42.648232 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.648187    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.648399 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.648323    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.648399 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.648357    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzt9w\" (UniqueName: \"kubernetes.io/projected/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-kube-api-access-nzt9w\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.748967 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.748881    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.748967 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.748927    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzt9w\" (UniqueName: \"kubernetes.io/projected/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-kube-api-access-nzt9w\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.749166 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.749011    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.749773 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.749742    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.751406 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.751375    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:42.805582 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:42.805561    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzt9w\" (UniqueName: \"kubernetes.io/projected/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-kube-api-access-nzt9w\") pod \"maas-api-revoke-sub-2e1f5b5f-psttb\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:43.098798 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:43.098700    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:43.473400 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:43.473367    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"]
Nov 03 17:05:43.474639 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:05:43.474610    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36fb9ce_3796_4130_9fdb_3e42f9a56a5b.slice/crio-1c730d686af13588706f1c85566b89a125f1f3fe37bb408d5565ec59a33f86a6 WatchSource:0}: Error finding container 1c730d686af13588706f1c85566b89a125f1f3fe37bb408d5565ec59a33f86a6: Status 404 returned error can't find the container with id 1c730d686af13588706f1c85566b89a125f1f3fe37bb408d5565ec59a33f86a6
Nov 03 17:05:43.693840 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:43.693806    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb" event={"ID":"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b","Type":"ContainerStarted","Data":"1c730d686af13588706f1c85566b89a125f1f3fe37bb408d5565ec59a33f86a6"}
Nov 03 17:05:44.700679 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:44.700641    5112 generic.go:356] "Generic (PLEG): container finished" podID="f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" containerID="e68324453045e37e15b26c8ce5485afcfea195d270a64b75c08e6a7fe45a5f29" exitCode=0
Nov 03 17:05:44.701067 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:44.700718    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb" event={"ID":"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b","Type":"ContainerDied","Data":"e68324453045e37e15b26c8ce5485afcfea195d270a64b75c08e6a7fe45a5f29"}
Nov 03 17:05:45.854655 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.854625    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:45.982447 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.982360    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-openshift-ca\") pod \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") "
Nov 03 17:05:45.982600 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.982498    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-kube-api-access-nzt9w\" (UniqueName: \"kubernetes.io/projected/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-kube-api-access-nzt9w\") pod \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") "
Nov 03 17:05:45.982600 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.982548    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-xks-ca\") pod \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\" (UID: \"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b\") "
Nov 03 17:05:45.982748 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.982721    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-openshift-ca" pod "f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" (UID: "f36fb9ce-3796-4130-9fdb-3e42f9a56a5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:05:45.982873 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.982855    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:45.984741 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.984713    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-kube-api-access-nzt9w" pod "f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" (UID: "f36fb9ce-3796-4130-9fdb-3e42f9a56a5b"). InnerVolumeSpecName "kube-api-access-nzt9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:05:45.984879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:45.984746    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-xks-ca" pod "f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" (UID: "f36fb9ce-3796-4130-9fdb-3e42f9a56a5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:05:46.084021 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:46.083985    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzt9w\" (UniqueName: \"kubernetes.io/projected/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-kube-api-access-nzt9w\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:46.084021 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:46.084019    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:46.713556 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:46.713520    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb" event={"ID":"f36fb9ce-3796-4130-9fdb-3e42f9a56a5b","Type":"ContainerDied","Data":"1c730d686af13588706f1c85566b89a125f1f3fe37bb408d5565ec59a33f86a6"}
Nov 03 17:05:46.713556 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:46.713549    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"
Nov 03 17:05:46.713780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:46.713571    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c730d686af13588706f1c85566b89a125f1f3fe37bb408d5565ec59a33f86a6"
Nov 03 17:05:55.136773 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.136733    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"]
Nov 03 17:05:55.137550 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.137527    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" containerName="revoke-keys"
Nov 03 17:05:55.137630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.137552    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" containerName="revoke-keys"
Nov 03 17:05:55.137680 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.137669    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" containerName="revoke-keys"
Nov 03 17:05:55.141804 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.141777    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.176508 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.176473    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.176718 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.176629    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.176718 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.176671    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxpq\" (UniqueName: \"kubernetes.io/projected/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-kube-api-access-pgxpq\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.197261 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.197228    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"]
Nov 03 17:05:55.240431 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.240401    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:05:55.240591 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.240454    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:05:55.278216 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.278168    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.278370 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.278301    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.278370 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.278333    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxpq\" (UniqueName: \"kubernetes.io/projected/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-kube-api-access-pgxpq\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.278961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.278940    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.280612 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.280590    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.334577 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.334547    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxpq\" (UniqueName: \"kubernetes.io/projected/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-kube-api-access-pgxpq\") pod \"maas-api-revoke-sub-ec843018-cs4cc\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.459508 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.459479    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:55.833987 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:55.833958    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"]
Nov 03 17:05:55.835900 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:05:55.835873    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb954bf_4ad5_41f0_ad19_766c9a2b3baa.slice/crio-edbfd6049b9fc1d6b508254dfb485229959d734d0dd360fdb01c74caabb80a5e WatchSource:0}: Error finding container edbfd6049b9fc1d6b508254dfb485229959d734d0dd360fdb01c74caabb80a5e: Status 404 returned error can't find the container with id edbfd6049b9fc1d6b508254dfb485229959d734d0dd360fdb01c74caabb80a5e
Nov 03 17:05:56.775397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:56.775354    5112 generic.go:356] "Generic (PLEG): container finished" podID="2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" containerID="2e9029b81026e74f43148ac3f762c0a6fad7b4816e55fb45a2d9d82ff967cfc5" exitCode=0
Nov 03 17:05:56.775835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:56.775420    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc" event={"ID":"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa","Type":"ContainerDied","Data":"2e9029b81026e74f43148ac3f762c0a6fad7b4816e55fb45a2d9d82ff967cfc5"}
Nov 03 17:05:56.775835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:56.775480    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc" event={"ID":"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa","Type":"ContainerStarted","Data":"edbfd6049b9fc1d6b508254dfb485229959d734d0dd360fdb01c74caabb80a5e"}
Nov 03 17:05:57.923149 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:57.923124    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:58.006533 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.006506    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-xks-ca\") pod \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") "
Nov 03 17:05:58.006683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.006549    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-kube-api-access-pgxpq\" (UniqueName: \"kubernetes.io/projected/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-kube-api-access-pgxpq\") pod \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") "
Nov 03 17:05:58.006683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.006639    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-openshift-ca\") pod \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\" (UID: \"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa\") "
Nov 03 17:05:58.007041 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.007014    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-openshift-ca" pod "2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" (UID: "2fb954bf-4ad5-41f0-ad19-766c9a2b3baa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:05:58.008762 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.008741    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-xks-ca" pod "2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" (UID: "2fb954bf-4ad5-41f0-ad19-766c9a2b3baa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:05:58.008826 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.008745    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-kube-api-access-pgxpq" pod "2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" (UID: "2fb954bf-4ad5-41f0-ad19-766c9a2b3baa"). InnerVolumeSpecName "kube-api-access-pgxpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:05:58.107559 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.107486    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:58.107559 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.107512    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:58.107559 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.107533    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgxpq\" (UniqueName: \"kubernetes.io/projected/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa-kube-api-access-pgxpq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:05:58.787469 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.787441    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"
Nov 03 17:05:58.787653 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.787481    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc" event={"ID":"2fb954bf-4ad5-41f0-ad19-766c9a2b3baa","Type":"ContainerDied","Data":"edbfd6049b9fc1d6b508254dfb485229959d734d0dd360fdb01c74caabb80a5e"}
Nov 03 17:05:58.787653 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:05:58.787537    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edbfd6049b9fc1d6b508254dfb485229959d734d0dd360fdb01c74caabb80a5e"
Nov 03 17:06:06.737695 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.737662    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"]
Nov 03 17:06:06.738154 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.738141    5112 cpu_manager.go:414] "containerMap: removing container" podUID="2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" containerName="revoke-keys"
Nov 03 17:06:06.738194 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.738155    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" containerName="revoke-keys"
Nov 03 17:06:06.738306 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.738291    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" containerName="revoke-keys"
Nov 03 17:06:06.743288 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.743262    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.795927 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.795891    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796124 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.795958    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkp98\" (UniqueName: \"kubernetes.io/projected/ffae2951-6be1-4108-9570-62531dccc289-kube-api-access-hkp98\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796124 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.796099    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffae2951-6be1-4108-9570-62531dccc289-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796242 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.796133    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796294 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.796276    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796342 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.796307    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffae2951-6be1-4108-9570-62531dccc289-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796342 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.796331    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796426 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.796382    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffae2951-6be1-4108-9570-62531dccc289-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.796426 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.796415    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.847424 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.847388    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"]
Nov 03 17:06:06.897030 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.896988    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffae2951-6be1-4108-9570-62531dccc289-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897030 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897034    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897268 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897217    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897317 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897280    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffae2951-6be1-4108-9570-62531dccc289-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897358 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897319    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897358 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897344    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffae2951-6be1-4108-9570-62531dccc289-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897465 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897383    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897465 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897423    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897571 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897523    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897662 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897523    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897755 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897696    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkp98\" (UniqueName: \"kubernetes.io/projected/ffae2951-6be1-4108-9570-62531dccc289-kube-api-access-hkp98\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897755 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897715    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.897910 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.897889    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.898089 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.898071    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffae2951-6be1-4108-9570-62531dccc289-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.899646 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.899623    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffae2951-6be1-4108-9570-62531dccc289-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.899754 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.899740    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffae2951-6be1-4108-9570-62531dccc289-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.954072 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.954028    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffae2951-6be1-4108-9570-62531dccc289-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:06.954283 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:06.954109    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkp98\" (UniqueName: \"kubernetes.io/projected/ffae2951-6be1-4108-9570-62531dccc289-kube-api-access-hkp98\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fgmxm\" (UID: \"ffae2951-6be1-4108-9570-62531dccc289\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:07.057336 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.057242    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:07.485103 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.485076    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"]
Nov 03 17:06:07.486637 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:06:07.486607    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffae2951_6be1_4108_9570_62531dccc289.slice/crio-97474ae89a09446f77043d13a4d5efba12047db4e417183b565bf9c1bbb9a7ce WatchSource:0}: Error finding container 97474ae89a09446f77043d13a4d5efba12047db4e417183b565bf9c1bbb9a7ce: Status 404 returned error can't find the container with id 97474ae89a09446f77043d13a4d5efba12047db4e417183b565bf9c1bbb9a7ce
Nov 03 17:06:07.489000 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.488968    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:06:07.489074 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.489042    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:06:07.489108 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.489070    5112 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 17:06:07.839320 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.839222    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" event={"ID":"ffae2951-6be1-4108-9570-62531dccc289","Type":"ContainerStarted","Data":"bc3a2b6aba2d05cb601b9f41e83e9c9070b2452bad2e18b11afb14a032810317"}
Nov 03 17:06:07.839320 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.839280    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" event={"ID":"ffae2951-6be1-4108-9570-62531dccc289","Type":"ContainerStarted","Data":"97474ae89a09446f77043d13a4d5efba12047db4e417183b565bf9c1bbb9a7ce"}
Nov 03 17:06:07.886363 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:06:07.886322    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:06:07.968223 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:07.962513    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podStartSLOduration=1.962492556 podStartE2EDuration="1.962492556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:06:07.958729909 +0000 UTC m=+3329.538033766" watchObservedRunningTime="2026-11-03 17:06:07.962492556 +0000 UTC m=+3329.541796413"
Nov 03 17:06:08.073225 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:08.066528    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:08.073225 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:08.066678    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:08.073225 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:08.066722    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:09.058634 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:09.058593    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:09.059254 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:09.059188    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:10.069664 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:10.069622    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:10.070304 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:10.070265    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:11.073222 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:11.072629    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:11.073222 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:11.072695    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:12.063685 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:12.063641    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:12.064468 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:12.063969    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:13.059223 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:13.058677    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:13.059223 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:13.058741    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:14.058143 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:14.058104    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:14.058511 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:14.058478    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:15.058262 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:15.058183    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 03 17:06:15.059104 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:15.059066    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 03 17:06:17.059631 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:17.059584    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:06:17.060212 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:17.059701    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:06:18.101520 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:18.101396    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:06:18.101520 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:18.101489    5112 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:06:18.473637 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:18.473601    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:18.474044 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:18.474021    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:18.480810 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:18.480786    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm"
Nov 03 17:06:37.882933 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:06:37.882895    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:06:41.808363 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.808312    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"]
Nov 03 17:06:41.814417 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.814392    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:41.864887 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.864858    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"]
Nov 03 17:06:41.906265 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.906237    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:41.906420 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.906307    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb9sm\" (UniqueName: \"kubernetes.io/projected/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-kube-api-access-qb9sm\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:41.906420 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.906396    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:41.908856 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.908832    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:06:41.908979 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:41.908878    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:06:42.007780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.007749    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb9sm\" (UniqueName: \"kubernetes.io/projected/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-kube-api-access-qb9sm\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:42.007780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.007782    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:42.007934 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.007842    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:42.008550 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.008532    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:42.010180 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.010156    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:42.064827 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.064770    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb9sm\" (UniqueName: \"kubernetes.io/projected/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-kube-api-access-qb9sm\") pod \"maas-api-revoke-sub-046e1343-xzd7p\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:42.127903 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.127881    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:42.501804 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:42.501775    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"]
Nov 03 17:06:42.503446 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:06:42.503415    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27cc0cb_b7a3_46d6_ae43_7948c6a2b9e0.slice/crio-4e258e0dadc085e3c8dad9c6889048ade2bdaea25845a104facbad0f2b76ba79 WatchSource:0}: Error finding container 4e258e0dadc085e3c8dad9c6889048ade2bdaea25845a104facbad0f2b76ba79: Status 404 returned error can't find the container with id 4e258e0dadc085e3c8dad9c6889048ade2bdaea25845a104facbad0f2b76ba79
Nov 03 17:06:43.080827 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:43.080744    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p" event={"ID":"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0","Type":"ContainerStarted","Data":"fa96d84cff1439919c981a95e1d917a51d3cad95162ccfe03adfedff57b902f8"}
Nov 03 17:06:43.081214 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:43.080833    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p" event={"ID":"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0","Type":"ContainerStarted","Data":"4e258e0dadc085e3c8dad9c6889048ade2bdaea25845a104facbad0f2b76ba79"}
Nov 03 17:06:43.194726 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:43.194670    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p" podStartSLOduration=2.194656627 podStartE2EDuration="2.194656627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:06:43.193797841 +0000 UTC m=+3364.773101727" watchObservedRunningTime="2026-11-03 17:06:43.194656627 +0000 UTC m=+3364.773960482"
Nov 03 17:06:44.088050 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:44.088017    5112 generic.go:356] "Generic (PLEG): container finished" podID="d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" containerID="fa96d84cff1439919c981a95e1d917a51d3cad95162ccfe03adfedff57b902f8" exitCode=0
Nov 03 17:06:44.088477 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:44.088096    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p" event={"ID":"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0","Type":"ContainerDied","Data":"fa96d84cff1439919c981a95e1d917a51d3cad95162ccfe03adfedff57b902f8"}
Nov 03 17:06:45.246613 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.246585    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:45.340465 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.340433    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-openshift-ca\") pod \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") "
Nov 03 17:06:45.340665 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.340500    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-xks-ca\") pod \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") "
Nov 03 17:06:45.340665 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.340658    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-kube-api-access-qb9sm\" (UniqueName: \"kubernetes.io/projected/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-kube-api-access-qb9sm\") pod \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\" (UID: \"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0\") "
Nov 03 17:06:45.340870 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.340845    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-openshift-ca" pod "d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" (UID: "d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:06:45.341004 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.340986    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:06:45.342683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.342646    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-xks-ca" pod "d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" (UID: "d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:06:45.342799 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.342710    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-kube-api-access-qb9sm" pod "d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" (UID: "d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0"). InnerVolumeSpecName "kube-api-access-qb9sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:06:45.442245 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.442189    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb9sm\" (UniqueName: \"kubernetes.io/projected/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-kube-api-access-qb9sm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:06:45.442245 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:45.442244    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:06:46.102264 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:46.102232    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"
Nov 03 17:06:46.102264 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:46.102241    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p" event={"ID":"d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0","Type":"ContainerDied","Data":"4e258e0dadc085e3c8dad9c6889048ade2bdaea25845a104facbad0f2b76ba79"}
Nov 03 17:06:46.102480 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:46.102294    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e258e0dadc085e3c8dad9c6889048ade2bdaea25845a104facbad0f2b76ba79"
Nov 03 17:06:52.642699 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.642664    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"]
Nov 03 17:06:52.643187 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.643153    5112 cpu_manager.go:414] "containerMap: removing container" podUID="d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" containerName="revoke-keys"
Nov 03 17:06:52.643187 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.643164    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" containerName="revoke-keys"
Nov 03 17:06:52.643707 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.643669    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" containerName="revoke-keys"
Nov 03 17:06:52.648507 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.648477    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.703861 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.703827    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"]
Nov 03 17:06:52.746899 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.746867    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:06:52.747053 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.746980    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:06:52.816076 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.816046    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.816309 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.816093    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrgd\" (UniqueName: \"kubernetes.io/projected/9871dc26-5ffe-4076-9061-d8732bda38db-kube-api-access-vlrgd\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.816309 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.816132    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.917663 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.917632    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.917822 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.917682    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrgd\" (UniqueName: \"kubernetes.io/projected/9871dc26-5ffe-4076-9061-d8732bda38db-kube-api-access-vlrgd\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.917822 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.917719    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.918404 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.918384    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.920084 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.920061    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:52.975269 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:52.975244    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrgd\" (UniqueName: \"kubernetes.io/projected/9871dc26-5ffe-4076-9061-d8732bda38db-kube-api-access-vlrgd\") pod \"maas-api-revoke-sub-d3fba056-q6kzg\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:53.266868 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:53.266785    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:53.645900 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:53.645869    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"]
Nov 03 17:06:53.646432 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:06:53.646405    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9871dc26_5ffe_4076_9061_d8732bda38db.slice/crio-f17d3621fe131e23619abd6092a9872756a830d1b9fcefb1cd6ecbd537b2d4ca WatchSource:0}: Error finding container f17d3621fe131e23619abd6092a9872756a830d1b9fcefb1cd6ecbd537b2d4ca: Status 404 returned error can't find the container with id f17d3621fe131e23619abd6092a9872756a830d1b9fcefb1cd6ecbd537b2d4ca
Nov 03 17:06:54.147023 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:54.146982    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg" event={"ID":"9871dc26-5ffe-4076-9061-d8732bda38db","Type":"ContainerStarted","Data":"0f8d3f27e9cf0753d6a423907f083afac5e3edfe93eb8b05d6ae0ac7407fca56"}
Nov 03 17:06:54.147248 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:54.147046    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg" event={"ID":"9871dc26-5ffe-4076-9061-d8732bda38db","Type":"ContainerStarted","Data":"f17d3621fe131e23619abd6092a9872756a830d1b9fcefb1cd6ecbd537b2d4ca"}
Nov 03 17:06:54.262694 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:54.262574    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg" podStartSLOduration=2.2625609 podStartE2EDuration="2.2625609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:06:54.259960891 +0000 UTC m=+3375.839264751" watchObservedRunningTime="2026-11-03 17:06:54.2625609 +0000 UTC m=+3375.841864754"
Nov 03 17:06:55.156636 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:55.156602    5112 generic.go:356] "Generic (PLEG): container finished" podID="9871dc26-5ffe-4076-9061-d8732bda38db" containerID="0f8d3f27e9cf0753d6a423907f083afac5e3edfe93eb8b05d6ae0ac7407fca56" exitCode=0
Nov 03 17:06:55.157238 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:55.156662    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg" event={"ID":"9871dc26-5ffe-4076-9061-d8732bda38db","Type":"ContainerDied","Data":"0f8d3f27e9cf0753d6a423907f083afac5e3edfe93eb8b05d6ae0ac7407fca56"}
Nov 03 17:06:56.777480 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.777440    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:06:56.869164 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.869127    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9871dc26-5ffe-4076-9061-d8732bda38db-kube-api-access-vlrgd\" (UniqueName: \"kubernetes.io/projected/9871dc26-5ffe-4076-9061-d8732bda38db-kube-api-access-vlrgd\") pod \"9871dc26-5ffe-4076-9061-d8732bda38db\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") "
Nov 03 17:06:56.869492 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.869465    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-openshift-ca\") pod \"9871dc26-5ffe-4076-9061-d8732bda38db\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") "
Nov 03 17:06:56.869707 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.869687    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-xks-ca\") pod \"9871dc26-5ffe-4076-9061-d8732bda38db\" (UID: \"9871dc26-5ffe-4076-9061-d8732bda38db\") "
Nov 03 17:06:56.870947 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.870922    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-openshift-ca" pod "9871dc26-5ffe-4076-9061-d8732bda38db" (UID: "9871dc26-5ffe-4076-9061-d8732bda38db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:06:56.877408 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.877378    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-xks-ca" pod "9871dc26-5ffe-4076-9061-d8732bda38db" (UID: "9871dc26-5ffe-4076-9061-d8732bda38db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:06:56.909207 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.909105    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9871dc26-5ffe-4076-9061-d8732bda38db-kube-api-access-vlrgd" pod "9871dc26-5ffe-4076-9061-d8732bda38db" (UID: "9871dc26-5ffe-4076-9061-d8732bda38db"). InnerVolumeSpecName "kube-api-access-vlrgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:06:56.971955 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.971916    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrgd\" (UniqueName: \"kubernetes.io/projected/9871dc26-5ffe-4076-9061-d8732bda38db-kube-api-access-vlrgd\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:06:56.978233 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.972233    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:06:56.995387 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:56.995344    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871dc26-5ffe-4076-9061-d8732bda38db-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:06:57.236454 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:57.236339    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg" event={"ID":"9871dc26-5ffe-4076-9061-d8732bda38db","Type":"ContainerDied","Data":"f17d3621fe131e23619abd6092a9872756a830d1b9fcefb1cd6ecbd537b2d4ca"}
Nov 03 17:06:57.236454 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:57.236418    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f17d3621fe131e23619abd6092a9872756a830d1b9fcefb1cd6ecbd537b2d4ca"
Nov 03 17:06:57.236708 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:06:57.236494    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"
Nov 03 17:07:07.890655 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:07:07.890604    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:07:27.403026 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.402985    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"]
Nov 03 17:07:27.403774 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.403751    5112 cpu_manager.go:414] "containerMap: removing container" podUID="9871dc26-5ffe-4076-9061-d8732bda38db" containerName="revoke-keys"
Nov 03 17:07:27.403889 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.403776    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9871dc26-5ffe-4076-9061-d8732bda38db" containerName="revoke-keys"
Nov 03 17:07:27.403946 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.403900    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="9871dc26-5ffe-4076-9061-d8732bda38db" containerName="revoke-keys"
Nov 03 17:07:27.407642 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.407618    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.449759 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.449719    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcz2z\" (UniqueName: \"kubernetes.io/projected/6c3329d1-f6ed-4628-a553-08ed45ddcffc-kube-api-access-zcz2z\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.449927 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.449786    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.450007 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.449911    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.462875 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.462843    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"]
Nov 03 17:07:27.506439 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.506408    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:07:27.506611 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.506486    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:07:27.550875 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.550835    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcz2z\" (UniqueName: \"kubernetes.io/projected/6c3329d1-f6ed-4628-a553-08ed45ddcffc-kube-api-access-zcz2z\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.551066 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.550922    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.551066 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.551006    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.551787 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.551761    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.553440 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.553418    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.608383 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.608351    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcz2z\" (UniqueName: \"kubernetes.io/projected/6c3329d1-f6ed-4628-a553-08ed45ddcffc-kube-api-access-zcz2z\") pod \"maas-api-revoke-sub-ae594979-rspzt\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:27.721955 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:27.721865    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:28.096163 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:28.096132    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"]
Nov 03 17:07:28.097072 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:07:28.097042    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3329d1_f6ed_4628_a553_08ed45ddcffc.slice/crio-cc3fe539f0598a337c297588e24ae9efd6968eac46ab77efe1b22d9b16f0987b WatchSource:0}: Error finding container cc3fe539f0598a337c297588e24ae9efd6968eac46ab77efe1b22d9b16f0987b: Status 404 returned error can't find the container with id cc3fe539f0598a337c297588e24ae9efd6968eac46ab77efe1b22d9b16f0987b
Nov 03 17:07:28.444085 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:28.444039    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt" event={"ID":"6c3329d1-f6ed-4628-a553-08ed45ddcffc","Type":"ContainerStarted","Data":"4ab2170431f10bf3c4efc68be9f2b261c25fc8dd9cf3792a2cd0b5a44d3bc1c0"}
Nov 03 17:07:28.444620 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:28.444112    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt" event={"ID":"6c3329d1-f6ed-4628-a553-08ed45ddcffc","Type":"ContainerStarted","Data":"cc3fe539f0598a337c297588e24ae9efd6968eac46ab77efe1b22d9b16f0987b"}
Nov 03 17:07:28.557774 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:28.557717    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt" podStartSLOduration=1.5577013339999999 podStartE2EDuration="1.557701334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:28.555298214 +0000 UTC m=+3410.134602087" watchObservedRunningTime="2026-11-03 17:07:28.557701334 +0000 UTC m=+3410.137005190"
Nov 03 17:07:29.451466 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:29.451431    5112 generic.go:356] "Generic (PLEG): container finished" podID="6c3329d1-f6ed-4628-a553-08ed45ddcffc" containerID="4ab2170431f10bf3c4efc68be9f2b261c25fc8dd9cf3792a2cd0b5a44d3bc1c0" exitCode=0
Nov 03 17:07:29.451893 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:29.451520    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt" event={"ID":"6c3329d1-f6ed-4628-a553-08ed45ddcffc","Type":"ContainerDied","Data":"4ab2170431f10bf3c4efc68be9f2b261c25fc8dd9cf3792a2cd0b5a44d3bc1c0"}
Nov 03 17:07:30.606091 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.606058    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:30.684263 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.684188    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-xks-ca\") pod \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") "
Nov 03 17:07:30.684490 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.684293    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3329d1-f6ed-4628-a553-08ed45ddcffc-kube-api-access-zcz2z\" (UniqueName: \"kubernetes.io/projected/6c3329d1-f6ed-4628-a553-08ed45ddcffc-kube-api-access-zcz2z\") pod \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") "
Nov 03 17:07:30.684490 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.684430    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-openshift-ca\") pod \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\" (UID: \"6c3329d1-f6ed-4628-a553-08ed45ddcffc\") "
Nov 03 17:07:30.684829 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.684800    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-openshift-ca" pod "6c3329d1-f6ed-4628-a553-08ed45ddcffc" (UID: "6c3329d1-f6ed-4628-a553-08ed45ddcffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:07:30.686389 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.686355    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-xks-ca" pod "6c3329d1-f6ed-4628-a553-08ed45ddcffc" (UID: "6c3329d1-f6ed-4628-a553-08ed45ddcffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:07:30.686487 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.686427    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3329d1-f6ed-4628-a553-08ed45ddcffc-kube-api-access-zcz2z" pod "6c3329d1-f6ed-4628-a553-08ed45ddcffc" (UID: "6c3329d1-f6ed-4628-a553-08ed45ddcffc"). InnerVolumeSpecName "kube-api-access-zcz2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:07:30.786260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.786138    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:07:30.786260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.786176    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3329d1-f6ed-4628-a553-08ed45ddcffc-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:07:30.786260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:30.786185    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcz2z\" (UniqueName: \"kubernetes.io/projected/6c3329d1-f6ed-4628-a553-08ed45ddcffc-kube-api-access-zcz2z\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:07:31.464055 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:31.464017    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"
Nov 03 17:07:31.464055 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:31.464046    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt" event={"ID":"6c3329d1-f6ed-4628-a553-08ed45ddcffc","Type":"ContainerDied","Data":"cc3fe539f0598a337c297588e24ae9efd6968eac46ab77efe1b22d9b16f0987b"}
Nov 03 17:07:31.464299 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:31.464124    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc3fe539f0598a337c297588e24ae9efd6968eac46ab77efe1b22d9b16f0987b"
Nov 03 17:07:37.883088 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:07:37.883047    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:07:55.511438 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.511339    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"]
Nov 03 17:07:55.512052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.511899    5112 cpu_manager.go:414] "containerMap: removing container" podUID="6c3329d1-f6ed-4628-a553-08ed45ddcffc" containerName="revoke-keys"
Nov 03 17:07:55.512052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.511914    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3329d1-f6ed-4628-a553-08ed45ddcffc" containerName="revoke-keys"
Nov 03 17:07:55.512052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.512005    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3329d1-f6ed-4628-a553-08ed45ddcffc" containerName="revoke-keys"
Nov 03 17:07:55.515299 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.515275    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.584612 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.584565    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"]
Nov 03 17:07:55.614350 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.614317    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:07:55.614543 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.614429    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:07:55.657958 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.657926    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.658147 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.658061    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.658236 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.658180    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpxx\" (UniqueName: \"kubernetes.io/projected/442dcb04-f937-4606-bed3-9b46b69618f3-kube-api-access-8xpxx\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.759105 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.759064    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.759339 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.759180    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpxx\" (UniqueName: \"kubernetes.io/projected/442dcb04-f937-4606-bed3-9b46b69618f3-kube-api-access-8xpxx\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.759339 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.759243    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.759902 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.759876    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.761632 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.761579    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.819065 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.819037    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpxx\" (UniqueName: \"kubernetes.io/projected/442dcb04-f937-4606-bed3-9b46b69618f3-kube-api-access-8xpxx\") pod \"maas-api-revoke-sub-a4aef7d7-4ch8f\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:55.831697 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:55.831671    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:56.207117 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:56.207070    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"]
Nov 03 17:07:56.207264 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:07:56.207070    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442dcb04_f937_4606_bed3_9b46b69618f3.slice/crio-53a69d7e76d8bd8e0822f71a624cbca366cfe9d8a2d3f0909106181c8ca74866 WatchSource:0}: Error finding container 53a69d7e76d8bd8e0822f71a624cbca366cfe9d8a2d3f0909106181c8ca74866: Status 404 returned error can't find the container with id 53a69d7e76d8bd8e0822f71a624cbca366cfe9d8a2d3f0909106181c8ca74866
Nov 03 17:07:56.670271 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:56.670235    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f" event={"ID":"442dcb04-f937-4606-bed3-9b46b69618f3","Type":"ContainerStarted","Data":"998854d363ef7588d320f60b92f57a581bf65f884f0db7d0a562c0c97cddd2b4"}
Nov 03 17:07:56.670271 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:56.670288    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f" event={"ID":"442dcb04-f937-4606-bed3-9b46b69618f3","Type":"ContainerStarted","Data":"53a69d7e76d8bd8e0822f71a624cbca366cfe9d8a2d3f0909106181c8ca74866"}
Nov 03 17:07:56.784452 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:56.784352    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f" podStartSLOduration=1.7843276270000001 podStartE2EDuration="1.784327627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:56.782905909 +0000 UTC m=+3438.362209764" watchObservedRunningTime="2026-11-03 17:07:56.784327627 +0000 UTC m=+3438.363631481"
Nov 03 17:07:57.677513 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:57.677480    5112 generic.go:356] "Generic (PLEG): container finished" podID="442dcb04-f937-4606-bed3-9b46b69618f3" containerID="998854d363ef7588d320f60b92f57a581bf65f884f0db7d0a562c0c97cddd2b4" exitCode=0
Nov 03 17:07:57.677905 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:57.677567    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f" event={"ID":"442dcb04-f937-4606-bed3-9b46b69618f3","Type":"ContainerDied","Data":"998854d363ef7588d320f60b92f57a581bf65f884f0db7d0a562c0c97cddd2b4"}
Nov 03 17:07:58.830307 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:58.830280    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:58.992310 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:58.992229    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-xks-ca\") pod \"442dcb04-f937-4606-bed3-9b46b69618f3\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") "
Nov 03 17:07:58.992310 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:58.992279    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442dcb04-f937-4606-bed3-9b46b69618f3-kube-api-access-8xpxx\" (UniqueName: \"kubernetes.io/projected/442dcb04-f937-4606-bed3-9b46b69618f3-kube-api-access-8xpxx\") pod \"442dcb04-f937-4606-bed3-9b46b69618f3\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") "
Nov 03 17:07:58.992536 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:58.992334    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-openshift-ca\") pod \"442dcb04-f937-4606-bed3-9b46b69618f3\" (UID: \"442dcb04-f937-4606-bed3-9b46b69618f3\") "
Nov 03 17:07:58.992771 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:58.992736    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-openshift-ca" pod "442dcb04-f937-4606-bed3-9b46b69618f3" (UID: "442dcb04-f937-4606-bed3-9b46b69618f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:07:58.994294 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:58.994273    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-xks-ca" pod "442dcb04-f937-4606-bed3-9b46b69618f3" (UID: "442dcb04-f937-4606-bed3-9b46b69618f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:07:58.994403 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:58.994383    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442dcb04-f937-4606-bed3-9b46b69618f3-kube-api-access-8xpxx" pod "442dcb04-f937-4606-bed3-9b46b69618f3" (UID: "442dcb04-f937-4606-bed3-9b46b69618f3"). InnerVolumeSpecName "kube-api-access-8xpxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:07:59.093698 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:59.093665    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:07:59.093698 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:59.093698    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xpxx\" (UniqueName: \"kubernetes.io/projected/442dcb04-f937-4606-bed3-9b46b69618f3-kube-api-access-8xpxx\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:07:59.093861 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:59.093712    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442dcb04-f937-4606-bed3-9b46b69618f3-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:07:59.690046 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:59.690019    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"
Nov 03 17:07:59.690227 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:59.690064    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f" event={"ID":"442dcb04-f937-4606-bed3-9b46b69618f3","Type":"ContainerDied","Data":"53a69d7e76d8bd8e0822f71a624cbca366cfe9d8a2d3f0909106181c8ca74866"}
Nov 03 17:07:59.690227 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:07:59.690110    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a69d7e76d8bd8e0822f71a624cbca366cfe9d8a2d3f0909106181c8ca74866"
Nov 03 17:08:07.885293 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:08:07.885250    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:08:26.088600 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.088561    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"]
Nov 03 17:08:26.089289 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.089268    5112 cpu_manager.go:414] "containerMap: removing container" podUID="442dcb04-f937-4606-bed3-9b46b69618f3" containerName="revoke-keys"
Nov 03 17:08:26.089413 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.089291    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442dcb04-f937-4606-bed3-9b46b69618f3" containerName="revoke-keys"
Nov 03 17:08:26.089413 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.089398    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="442dcb04-f937-4606-bed3-9b46b69618f3" containerName="revoke-keys"
Nov 03 17:08:26.092754 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.092730    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.148401 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.148371    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"]
Nov 03 17:08:26.191838 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.191806    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:08:26.192012 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.191868    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:08:26.227390 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.227358    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.227621 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.227530    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.227621 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.227586    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn42p\" (UniqueName: \"kubernetes.io/projected/d0597c33-f90c-4c71-b7bc-3cbaa419da26-kube-api-access-kn42p\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.328398 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.328362    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.328398 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.328405    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn42p\" (UniqueName: \"kubernetes.io/projected/d0597c33-f90c-4c71-b7bc-3cbaa419da26-kube-api-access-kn42p\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.328617 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.328463    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.329067 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.329030    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.330892 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.330868    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.385485 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.385416    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn42p\" (UniqueName: \"kubernetes.io/projected/d0597c33-f90c-4c71-b7bc-3cbaa419da26-kube-api-access-kn42p\") pod \"maas-api-revoke-sub-79805ca9-z9wp4\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.415011 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.414971    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:26.791034 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.790987    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"]
Nov 03 17:08:26.792434 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:08:26.792404    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0597c33_f90c_4c71_b7bc_3cbaa419da26.slice/crio-de8e1fb10a76e63fffc69590af6f2ef66a0fba5295294da2567bca614114cb2b WatchSource:0}: Error finding container de8e1fb10a76e63fffc69590af6f2ef66a0fba5295294da2567bca614114cb2b: Status 404 returned error can't find the container with id de8e1fb10a76e63fffc69590af6f2ef66a0fba5295294da2567bca614114cb2b
Nov 03 17:08:26.884120 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:26.884072    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4" event={"ID":"d0597c33-f90c-4c71-b7bc-3cbaa419da26","Type":"ContainerStarted","Data":"de8e1fb10a76e63fffc69590af6f2ef66a0fba5295294da2567bca614114cb2b"}
Nov 03 17:08:27.890978 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:27.890900    5112 generic.go:356] "Generic (PLEG): container finished" podID="d0597c33-f90c-4c71-b7bc-3cbaa419da26" containerID="b255a2192b04450443f31cce701ff2b7058362c43b1f031dd6f0670760c0bcc0" exitCode=0
Nov 03 17:08:27.891330 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:27.890987    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4" event={"ID":"d0597c33-f90c-4c71-b7bc-3cbaa419da26","Type":"ContainerDied","Data":"b255a2192b04450443f31cce701ff2b7058362c43b1f031dd6f0670760c0bcc0"}
Nov 03 17:08:29.038120 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.038096    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:29.157807 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.157726    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-xks-ca\") pod \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") "
Nov 03 17:08:29.157977 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.157826    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0597c33-f90c-4c71-b7bc-3cbaa419da26-kube-api-access-kn42p\" (UniqueName: \"kubernetes.io/projected/d0597c33-f90c-4c71-b7bc-3cbaa419da26-kube-api-access-kn42p\") pod \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") "
Nov 03 17:08:29.157977 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.157872    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-openshift-ca\") pod \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\" (UID: \"d0597c33-f90c-4c71-b7bc-3cbaa419da26\") "
Nov 03 17:08:29.158490 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.158457    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-openshift-ca" pod "d0597c33-f90c-4c71-b7bc-3cbaa419da26" (UID: "d0597c33-f90c-4c71-b7bc-3cbaa419da26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:08:29.159995 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.159964    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-xks-ca" pod "d0597c33-f90c-4c71-b7bc-3cbaa419da26" (UID: "d0597c33-f90c-4c71-b7bc-3cbaa419da26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:08:29.160124 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.160016    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0597c33-f90c-4c71-b7bc-3cbaa419da26-kube-api-access-kn42p" pod "d0597c33-f90c-4c71-b7bc-3cbaa419da26" (UID: "d0597c33-f90c-4c71-b7bc-3cbaa419da26"). InnerVolumeSpecName "kube-api-access-kn42p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:08:29.258689 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.258662    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:08:29.258689 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.258686    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn42p\" (UniqueName: \"kubernetes.io/projected/d0597c33-f90c-4c71-b7bc-3cbaa419da26-kube-api-access-kn42p\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:08:29.258844 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.258697    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0597c33-f90c-4c71-b7bc-3cbaa419da26-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:08:29.902796 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.902769    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"
Nov 03 17:08:29.902976 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.902812    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4" event={"ID":"d0597c33-f90c-4c71-b7bc-3cbaa419da26","Type":"ContainerDied","Data":"de8e1fb10a76e63fffc69590af6f2ef66a0fba5295294da2567bca614114cb2b"}
Nov 03 17:08:29.902976 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:29.902865    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de8e1fb10a76e63fffc69590af6f2ef66a0fba5295294da2567bca614114cb2b"
Nov 03 17:08:37.885746 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:08:37.885701    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:08:49.857344 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.857308    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"]
Nov 03 17:08:49.857861 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.857845    5112 cpu_manager.go:414] "containerMap: removing container" podUID="d0597c33-f90c-4c71-b7bc-3cbaa419da26" containerName="revoke-keys"
Nov 03 17:08:49.857907 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.857862    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0597c33-f90c-4c71-b7bc-3cbaa419da26" containerName="revoke-keys"
Nov 03 17:08:49.857959 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.857950    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0597c33-f90c-4c71-b7bc-3cbaa419da26" containerName="revoke-keys"
Nov 03 17:08:49.861475 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.861451    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:49.901966 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.901929    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:49.901966 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.901964    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:49.902174 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.902077    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjsqh\" (UniqueName: \"kubernetes.io/projected/bbf50353-b8f4-40ed-9028-00cc902c155d-kube-api-access-mjsqh\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:49.917300 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.917265    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"]
Nov 03 17:08:49.960753 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.960725    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:08:49.960914 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:49.960814    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:08:50.003644 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.003615    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:50.003797 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.003651    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:50.003797 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.003724    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjsqh\" (UniqueName: \"kubernetes.io/projected/bbf50353-b8f4-40ed-9028-00cc902c155d-kube-api-access-mjsqh\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:50.004264 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.004244    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:50.006091 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.006071    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:50.061531 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.061503    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjsqh\" (UniqueName: \"kubernetes.io/projected/bbf50353-b8f4-40ed-9028-00cc902c155d-kube-api-access-mjsqh\") pod \"maas-api-revoke-sub-0b7e8edb-nsw6k\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:50.174861 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.174835    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:50.548398 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:08:50.548366    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf50353_b8f4_40ed_9028_00cc902c155d.slice/crio-9160b61368ead8783c8f4fbff87198d0806d3904c6f0ac9420293f3656e4ed50 WatchSource:0}: Error finding container 9160b61368ead8783c8f4fbff87198d0806d3904c6f0ac9420293f3656e4ed50: Status 404 returned error can't find the container with id 9160b61368ead8783c8f4fbff87198d0806d3904c6f0ac9420293f3656e4ed50
Nov 03 17:08:50.550919 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:50.550893    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"]
Nov 03 17:08:51.064972 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:51.064937    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k" event={"ID":"bbf50353-b8f4-40ed-9028-00cc902c155d","Type":"ContainerStarted","Data":"0f266a333c5787b5fde9983ee0f2f0fe1adfcc420349a0ef842aa82f0c9791e7"}
Nov 03 17:08:51.065318 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:51.064986    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k" event={"ID":"bbf50353-b8f4-40ed-9028-00cc902c155d","Type":"ContainerStarted","Data":"9160b61368ead8783c8f4fbff87198d0806d3904c6f0ac9420293f3656e4ed50"}
Nov 03 17:08:51.178573 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:51.178517    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k" podStartSLOduration=2.178504327 podStartE2EDuration="2.178504327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:08:51.17595379 +0000 UTC m=+3492.755257664" watchObservedRunningTime="2026-11-03 17:08:51.178504327 +0000 UTC m=+3492.757808183"
Nov 03 17:08:52.072791 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:52.072755    5112 generic.go:356] "Generic (PLEG): container finished" podID="bbf50353-b8f4-40ed-9028-00cc902c155d" containerID="0f266a333c5787b5fde9983ee0f2f0fe1adfcc420349a0ef842aa82f0c9791e7" exitCode=0
Nov 03 17:08:52.073192 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:52.072797    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k" event={"ID":"bbf50353-b8f4-40ed-9028-00cc902c155d","Type":"ContainerDied","Data":"0f266a333c5787b5fde9983ee0f2f0fe1adfcc420349a0ef842aa82f0c9791e7"}
Nov 03 17:08:53.244660 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.244628    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:53.334904 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.334802    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf50353-b8f4-40ed-9028-00cc902c155d-kube-api-access-mjsqh\" (UniqueName: \"kubernetes.io/projected/bbf50353-b8f4-40ed-9028-00cc902c155d-kube-api-access-mjsqh\") pod \"bbf50353-b8f4-40ed-9028-00cc902c155d\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") "
Nov 03 17:08:53.334904 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.334856    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-openshift-ca\") pod \"bbf50353-b8f4-40ed-9028-00cc902c155d\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") "
Nov 03 17:08:53.334904 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.334885    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-xks-ca\") pod \"bbf50353-b8f4-40ed-9028-00cc902c155d\" (UID: \"bbf50353-b8f4-40ed-9028-00cc902c155d\") "
Nov 03 17:08:53.335312 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.335284    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-openshift-ca" pod "bbf50353-b8f4-40ed-9028-00cc902c155d" (UID: "bbf50353-b8f4-40ed-9028-00cc902c155d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:08:53.337076 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.337053    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-xks-ca" pod "bbf50353-b8f4-40ed-9028-00cc902c155d" (UID: "bbf50353-b8f4-40ed-9028-00cc902c155d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:08:53.337148 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.337115    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf50353-b8f4-40ed-9028-00cc902c155d-kube-api-access-mjsqh" pod "bbf50353-b8f4-40ed-9028-00cc902c155d" (UID: "bbf50353-b8f4-40ed-9028-00cc902c155d"). InnerVolumeSpecName "kube-api-access-mjsqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:08:53.435809 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.435771    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjsqh\" (UniqueName: \"kubernetes.io/projected/bbf50353-b8f4-40ed-9028-00cc902c155d-kube-api-access-mjsqh\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:08:53.435809 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.435800    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:08:53.435809 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:53.435810    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf50353-b8f4-40ed-9028-00cc902c155d-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:08:54.085032 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:54.085002    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"
Nov 03 17:08:54.085226 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:54.085037    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k" event={"ID":"bbf50353-b8f4-40ed-9028-00cc902c155d","Type":"ContainerDied","Data":"9160b61368ead8783c8f4fbff87198d0806d3904c6f0ac9420293f3656e4ed50"}
Nov 03 17:08:54.085226 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:54.085085    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9160b61368ead8783c8f4fbff87198d0806d3904c6f0ac9420293f3656e4ed50"
Nov 03 17:08:56.058121 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:56.058087    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"]
Nov 03 17:08:56.105010 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:56.104978    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7247h"]
Nov 03 17:08:56.549270 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:08:56.549232    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014a8a9d-5a65-4f79-8303-a24f771c97ea" path="/var/lib/kubelet/pods/014a8a9d-5a65-4f79-8303-a24f771c97ea/volumes"
Nov 03 17:09:01.209806 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.209771    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"]
Nov 03 17:09:01.210311 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.210295    5112 cpu_manager.go:414] "containerMap: removing container" podUID="bbf50353-b8f4-40ed-9028-00cc902c155d" containerName="revoke-keys"
Nov 03 17:09:01.210368 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.210311    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf50353-b8f4-40ed-9028-00cc902c155d" containerName="revoke-keys"
Nov 03 17:09:01.210409 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.210384    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf50353-b8f4-40ed-9028-00cc902c155d" containerName="revoke-keys"
Nov 03 17:09:01.214742 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.214721    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.267475 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.267448    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:09:01.267624 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.267492    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:09:01.269806 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.269781    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"]
Nov 03 17:09:01.313155 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.313120    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.313316 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.313189    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-525jj\" (UniqueName: \"kubernetes.io/projected/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-kube-api-access-525jj\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.313375 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.313352    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.414029 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.413995    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.414269 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.414063    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.414269 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.414156    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-525jj\" (UniqueName: \"kubernetes.io/projected/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-kube-api-access-525jj\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.414729 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.414706    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.416628 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.416606    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.471073 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.471000    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-525jj\" (UniqueName: \"kubernetes.io/projected/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-kube-api-access-525jj\") pod \"maas-api-revoke-sub-3d4156f1-4jbxx\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.527344 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.527308    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:01.899494 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:01.899460    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"]
Nov 03 17:09:01.900631 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:09:01.900603    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b3d4a3_3f54_497d_b97d_d0cb3138afbd.slice/crio-3ac9141d9fcaf455953a6857efed485a8172359d9ed33f659a61f31d4b7f2d0f WatchSource:0}: Error finding container 3ac9141d9fcaf455953a6857efed485a8172359d9ed33f659a61f31d4b7f2d0f: Status 404 returned error can't find the container with id 3ac9141d9fcaf455953a6857efed485a8172359d9ed33f659a61f31d4b7f2d0f
Nov 03 17:09:02.131852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:02.131814    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx" event={"ID":"57b3d4a3-3f54-497d-b97d-d0cb3138afbd","Type":"ContainerStarted","Data":"3ac9141d9fcaf455953a6857efed485a8172359d9ed33f659a61f31d4b7f2d0f"}
Nov 03 17:09:03.161087 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:03.161056    5112 generic.go:356] "Generic (PLEG): container finished" podID="57b3d4a3-3f54-497d-b97d-d0cb3138afbd" containerID="08ec9c96deb9f3f46fce8505495755b41384928f40160502802736a8233f549e" exitCode=0
Nov 03 17:09:03.161550 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:03.161164    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx" event={"ID":"57b3d4a3-3f54-497d-b97d-d0cb3138afbd","Type":"ContainerDied","Data":"08ec9c96deb9f3f46fce8505495755b41384928f40160502802736a8233f549e"}
Nov 03 17:09:04.536762 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.533354    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fgmxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:09:04.536762 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.533424    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fgmxm" podUID="ffae2951-6be1-4108-9570-62531dccc289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:09:04.730225 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.729464    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:04.760222 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.758566    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-openshift-ca\") pod \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") "
Nov 03 17:09:04.760222 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.758653    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-xks-ca\") pod \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") "
Nov 03 17:09:04.760222 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.758700    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-kube-api-access-525jj\" (UniqueName: \"kubernetes.io/projected/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-kube-api-access-525jj\") pod \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\" (UID: \"57b3d4a3-3f54-497d-b97d-d0cb3138afbd\") "
Nov 03 17:09:04.765220 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.764527    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-openshift-ca" pod "57b3d4a3-3f54-497d-b97d-d0cb3138afbd" (UID: "57b3d4a3-3f54-497d-b97d-d0cb3138afbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:09:04.775072 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.775042    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-xks-ca" pod "57b3d4a3-3f54-497d-b97d-d0cb3138afbd" (UID: "57b3d4a3-3f54-497d-b97d-d0cb3138afbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:09:04.793269 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.792406    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-kube-api-access-525jj" pod "57b3d4a3-3f54-497d-b97d-d0cb3138afbd" (UID: "57b3d4a3-3f54-497d-b97d-d0cb3138afbd"). InnerVolumeSpecName "kube-api-access-525jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:09:04.859886 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.859845    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:09:04.860133 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.860112    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:09:04.860275 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:04.860257    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-525jj\" (UniqueName: \"kubernetes.io/projected/57b3d4a3-3f54-497d-b97d-d0cb3138afbd-kube-api-access-525jj\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:09:05.251059 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:05.251021    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"
Nov 03 17:09:05.251281 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:05.251023    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx" event={"ID":"57b3d4a3-3f54-497d-b97d-d0cb3138afbd","Type":"ContainerDied","Data":"3ac9141d9fcaf455953a6857efed485a8172359d9ed33f659a61f31d4b7f2d0f"}
Nov 03 17:09:05.251281 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:05.251178    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ac9141d9fcaf455953a6857efed485a8172359d9ed33f659a61f31d4b7f2d0f"
Nov 03 17:09:07.882869 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:09:07.882828    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:09:31.739792 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.739757    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"]
Nov 03 17:09:31.787044 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.787011    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4jbxx"]
Nov 03 17:09:31.879153 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.879121    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"]
Nov 03 17:09:31.879616 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.879602    5112 cpu_manager.go:414] "containerMap: removing container" podUID="57b3d4a3-3f54-497d-b97d-d0cb3138afbd" containerName="revoke-keys"
Nov 03 17:09:31.879669 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.879616    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b3d4a3-3f54-497d-b97d-d0cb3138afbd" containerName="revoke-keys"
Nov 03 17:09:31.879725 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.879716    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b3d4a3-3f54-497d-b97d-d0cb3138afbd" containerName="revoke-keys"
Nov 03 17:09:31.882675 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.882658    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:31.937260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.937217    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:09:31.937481 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.937400    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:09:31.940403 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:31.940377    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"]
Nov 03 17:09:32.010852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.010743    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.010852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.010780    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.010852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.010803    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdf6\" (UniqueName: \"kubernetes.io/projected/3c622aa9-2161-43d8-9378-53eac7081833-kube-api-access-qhdf6\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.111758 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.111720    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.111758 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.111759    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.112037 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.111782    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdf6\" (UniqueName: \"kubernetes.io/projected/3c622aa9-2161-43d8-9378-53eac7081833-kube-api-access-qhdf6\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.112578 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.112554    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.114151 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.114131    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.170778 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.170747    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdf6\" (UniqueName: \"kubernetes.io/projected/3c622aa9-2161-43d8-9378-53eac7081833-kube-api-access-qhdf6\") pod \"maas-api-revoke-sub-3d4156f1-q6ml9\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.197369 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.197334    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:32.555183 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.555098    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b3d4a3-3f54-497d-b97d-d0cb3138afbd" path="/var/lib/kubelet/pods/57b3d4a3-3f54-497d-b97d-d0cb3138afbd/volumes"
Nov 03 17:09:32.572321 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:32.572297    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"]
Nov 03 17:09:32.573998 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:09:32.573973    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c622aa9_2161_43d8_9378_53eac7081833.slice/crio-5b537e348901a01c83cd55f6350b5e23277ecfdf6799d9cc08053e3e760345e7 WatchSource:0}: Error finding container 5b537e348901a01c83cd55f6350b5e23277ecfdf6799d9cc08053e3e760345e7: Status 404 returned error can't find the container with id 5b537e348901a01c83cd55f6350b5e23277ecfdf6799d9cc08053e3e760345e7
Nov 03 17:09:33.450259 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:33.450232    5112 generic.go:356] "Generic (PLEG): container finished" podID="3c622aa9-2161-43d8-9378-53eac7081833" containerID="54dd6ab260368386aefeea36d626d65a97d563413c1c521cde0471f2e310ada6" exitCode=0
Nov 03 17:09:33.450595 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:33.450315    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9" event={"ID":"3c622aa9-2161-43d8-9378-53eac7081833","Type":"ContainerDied","Data":"54dd6ab260368386aefeea36d626d65a97d563413c1c521cde0471f2e310ada6"}
Nov 03 17:09:33.450595 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:33.450378    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9" event={"ID":"3c622aa9-2161-43d8-9378-53eac7081833","Type":"ContainerStarted","Data":"5b537e348901a01c83cd55f6350b5e23277ecfdf6799d9cc08053e3e760345e7"}
Nov 03 17:09:34.599437 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.599400    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:34.743220 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.743129    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c622aa9-2161-43d8-9378-53eac7081833-kube-api-access-qhdf6\" (UniqueName: \"kubernetes.io/projected/3c622aa9-2161-43d8-9378-53eac7081833-kube-api-access-qhdf6\") pod \"3c622aa9-2161-43d8-9378-53eac7081833\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") "
Nov 03 17:09:34.743392 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.743229    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-openshift-ca\") pod \"3c622aa9-2161-43d8-9378-53eac7081833\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") "
Nov 03 17:09:34.743392 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.743353    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-xks-ca\") pod \"3c622aa9-2161-43d8-9378-53eac7081833\" (UID: \"3c622aa9-2161-43d8-9378-53eac7081833\") "
Nov 03 17:09:34.743682 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.743654    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-openshift-ca" pod "3c622aa9-2161-43d8-9378-53eac7081833" (UID: "3c622aa9-2161-43d8-9378-53eac7081833"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:09:34.745445 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.745413    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c622aa9-2161-43d8-9378-53eac7081833-kube-api-access-qhdf6" pod "3c622aa9-2161-43d8-9378-53eac7081833" (UID: "3c622aa9-2161-43d8-9378-53eac7081833"). InnerVolumeSpecName "kube-api-access-qhdf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:09:34.745553 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.745461    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-xks-ca" pod "3c622aa9-2161-43d8-9378-53eac7081833" (UID: "3c622aa9-2161-43d8-9378-53eac7081833"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:09:34.844780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.844741    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:09:34.844780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.844772    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdf6\" (UniqueName: \"kubernetes.io/projected/3c622aa9-2161-43d8-9378-53eac7081833-kube-api-access-qhdf6\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:09:34.844780 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:34.844783    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c622aa9-2161-43d8-9378-53eac7081833-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:09:35.463666 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:35.463639    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"
Nov 03 17:09:35.463860 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:35.463667    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9" event={"ID":"3c622aa9-2161-43d8-9378-53eac7081833","Type":"ContainerDied","Data":"5b537e348901a01c83cd55f6350b5e23277ecfdf6799d9cc08053e3e760345e7"}
Nov 03 17:09:35.463860 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:09:35.463716    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b537e348901a01c83cd55f6350b5e23277ecfdf6799d9cc08053e3e760345e7"
Nov 03 17:09:37.882150 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:09:37.882115    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:10:07.883313 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:10:07.883275    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:10:11.090567 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:11.090532    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"]
Nov 03 17:10:11.107259 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:11.107228    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-w9t7l"]
Nov 03 17:10:12.552814 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:12.552777    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d3a30d-7196-4142-8f4a-90ac9273ac2e" path="/var/lib/kubelet/pods/61d3a30d-7196-4142-8f4a-90ac9273ac2e/volumes"
Nov 03 17:10:22.036751 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.036719    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"]
Nov 03 17:10:22.037254 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.037239    5112 cpu_manager.go:414] "containerMap: removing container" podUID="3c622aa9-2161-43d8-9378-53eac7081833" containerName="revoke-keys"
Nov 03 17:10:22.037254 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.037255    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c622aa9-2161-43d8-9378-53eac7081833" containerName="revoke-keys"
Nov 03 17:10:22.037362 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.037337    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c622aa9-2161-43d8-9378-53eac7081833" containerName="revoke-keys"
Nov 03 17:10:22.040692 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.040675    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.096764 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.096731    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"]
Nov 03 17:10:22.126926 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.126894    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrwr\" (UniqueName: \"kubernetes.io/projected/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-kube-api-access-5lrwr\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.127106 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.127045    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.127106 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.127098    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.140605 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.140575    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:10:22.140749 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.140649    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:10:22.228360 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.228323    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.228571 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.228370    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.228571 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.228413    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrwr\" (UniqueName: \"kubernetes.io/projected/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-kube-api-access-5lrwr\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.229124 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.229089    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.235096 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.235069    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.284826 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.284787    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrwr\" (UniqueName: \"kubernetes.io/projected/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-kube-api-access-5lrwr\") pod \"maas-api-revoke-sub-e2fc9c5b-h8fvq\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.354999 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.354899    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:22.728670 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.728633    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"]
Nov 03 17:10:22.729422 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:10:22.729395    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d6cb8c_0346_4df7_92e2_b2c2507d5801.slice/crio-c5026b96cf3ef605e3a142a305ddc779351b868ab8450d298eb05af8f054e3d5 WatchSource:0}: Error finding container c5026b96cf3ef605e3a142a305ddc779351b868ab8450d298eb05af8f054e3d5: Status 404 returned error can't find the container with id c5026b96cf3ef605e3a142a305ddc779351b868ab8450d298eb05af8f054e3d5
Nov 03 17:10:22.777155 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:22.777120    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq" event={"ID":"f1d6cb8c-0346-4df7-92e2-b2c2507d5801","Type":"ContainerStarted","Data":"c5026b96cf3ef605e3a142a305ddc779351b868ab8450d298eb05af8f054e3d5"}
Nov 03 17:10:23.784737 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:23.784702    5112 generic.go:356] "Generic (PLEG): container finished" podID="f1d6cb8c-0346-4df7-92e2-b2c2507d5801" containerID="7be1757afae265cfe67c44966fc9eb10720d33522dac2d55b22bf56f1d8ffb72" exitCode=0
Nov 03 17:10:23.785132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:23.784769    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq" event={"ID":"f1d6cb8c-0346-4df7-92e2-b2c2507d5801","Type":"ContainerDied","Data":"7be1757afae265cfe67c44966fc9eb10720d33522dac2d55b22bf56f1d8ffb72"}
Nov 03 17:10:24.933082 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:24.933051    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:25.068760 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.068679    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-xks-ca\") pod \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") "
Nov 03 17:10:25.068930 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.068790    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-kube-api-access-5lrwr\" (UniqueName: \"kubernetes.io/projected/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-kube-api-access-5lrwr\") pod \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") "
Nov 03 17:10:25.068930 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.068903    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-openshift-ca\") pod \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\" (UID: \"f1d6cb8c-0346-4df7-92e2-b2c2507d5801\") "
Nov 03 17:10:25.069318 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.069289    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-openshift-ca" pod "f1d6cb8c-0346-4df7-92e2-b2c2507d5801" (UID: "f1d6cb8c-0346-4df7-92e2-b2c2507d5801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:10:25.070968 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.070943    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-kube-api-access-5lrwr" pod "f1d6cb8c-0346-4df7-92e2-b2c2507d5801" (UID: "f1d6cb8c-0346-4df7-92e2-b2c2507d5801"). InnerVolumeSpecName "kube-api-access-5lrwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:10:25.071287 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.071258    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-xks-ca" pod "f1d6cb8c-0346-4df7-92e2-b2c2507d5801" (UID: "f1d6cb8c-0346-4df7-92e2-b2c2507d5801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:10:25.170719 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.170678    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:25.170719 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.170717    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:25.170932 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.170734    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrwr\" (UniqueName: \"kubernetes.io/projected/f1d6cb8c-0346-4df7-92e2-b2c2507d5801-kube-api-access-5lrwr\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:25.797745 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.797710    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq" event={"ID":"f1d6cb8c-0346-4df7-92e2-b2c2507d5801","Type":"ContainerDied","Data":"c5026b96cf3ef605e3a142a305ddc779351b868ab8450d298eb05af8f054e3d5"}
Nov 03 17:10:25.797934 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.797763    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5026b96cf3ef605e3a142a305ddc779351b868ab8450d298eb05af8f054e3d5"
Nov 03 17:10:25.797934 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:25.797718    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"
Nov 03 17:10:32.867522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.867489    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"]
Nov 03 17:10:32.868018 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.868005    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f1d6cb8c-0346-4df7-92e2-b2c2507d5801" containerName="revoke-keys"
Nov 03 17:10:32.868018 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.868020    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d6cb8c-0346-4df7-92e2-b2c2507d5801" containerName="revoke-keys"
Nov 03 17:10:32.868145 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.868134    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d6cb8c-0346-4df7-92e2-b2c2507d5801" containerName="revoke-keys"
Nov 03 17:10:32.871467 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.871437    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:32.927827 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.927794    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"]
Nov 03 17:10:32.970535 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.970504    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:10:32.970706 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:32.970616    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:10:33.048389 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.048356    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.048389 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.048389    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.048619 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.048487    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfttc\" (UniqueName: \"kubernetes.io/projected/bc1b0625-2cb4-4cae-94bc-95148306ea08-kube-api-access-mfttc\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.149835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.149801    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfttc\" (UniqueName: \"kubernetes.io/projected/bc1b0625-2cb4-4cae-94bc-95148306ea08-kube-api-access-mfttc\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.150004 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.149924    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.150004 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.149950    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.150740 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.150714    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.152500 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.152474    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.207634 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.207593    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfttc\" (UniqueName: \"kubernetes.io/projected/bc1b0625-2cb4-4cae-94bc-95148306ea08-kube-api-access-mfttc\") pod \"maas-api-revoke-sub-5a29efb3-dtc8z\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.497957 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.497879    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:33.876770 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:33.876716    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"]
Nov 03 17:10:33.880569 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:10:33.880541    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b0625_2cb4_4cae_94bc_95148306ea08.slice/crio-21f2839340a231d3adbc39017fd9b13191e2931572527061849b2afc1858a61f WatchSource:0}: Error finding container 21f2839340a231d3adbc39017fd9b13191e2931572527061849b2afc1858a61f: Status 404 returned error can't find the container with id 21f2839340a231d3adbc39017fd9b13191e2931572527061849b2afc1858a61f
Nov 03 17:10:34.864332 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:34.864290    5112 generic.go:356] "Generic (PLEG): container finished" podID="bc1b0625-2cb4-4cae-94bc-95148306ea08" containerID="def1449210cb9aa612a1012bde47f2a97a5550560a174bf67c3d9bfcb1897cc5" exitCode=0
Nov 03 17:10:34.864508 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:34.864375    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z" event={"ID":"bc1b0625-2cb4-4cae-94bc-95148306ea08","Type":"ContainerDied","Data":"def1449210cb9aa612a1012bde47f2a97a5550560a174bf67c3d9bfcb1897cc5"}
Nov 03 17:10:34.864508 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:34.864451    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z" event={"ID":"bc1b0625-2cb4-4cae-94bc-95148306ea08","Type":"ContainerStarted","Data":"21f2839340a231d3adbc39017fd9b13191e2931572527061849b2afc1858a61f"}
Nov 03 17:10:36.040766 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.040733    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:36.198675 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.198619    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1b0625-2cb4-4cae-94bc-95148306ea08-kube-api-access-mfttc\" (UniqueName: \"kubernetes.io/projected/bc1b0625-2cb4-4cae-94bc-95148306ea08-kube-api-access-mfttc\") pod \"bc1b0625-2cb4-4cae-94bc-95148306ea08\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") "
Nov 03 17:10:36.199052 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.199026    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-xks-ca\") pod \"bc1b0625-2cb4-4cae-94bc-95148306ea08\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") "
Nov 03 17:10:36.203410 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.203382    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-openshift-ca\") pod \"bc1b0625-2cb4-4cae-94bc-95148306ea08\" (UID: \"bc1b0625-2cb4-4cae-94bc-95148306ea08\") "
Nov 03 17:10:36.242717 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.242678    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-openshift-ca" pod "bc1b0625-2cb4-4cae-94bc-95148306ea08" (UID: "bc1b0625-2cb4-4cae-94bc-95148306ea08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:10:36.243101 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.243077    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b0625-2cb4-4cae-94bc-95148306ea08-kube-api-access-mfttc" pod "bc1b0625-2cb4-4cae-94bc-95148306ea08" (UID: "bc1b0625-2cb4-4cae-94bc-95148306ea08"). InnerVolumeSpecName "kube-api-access-mfttc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:10:36.244332 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.244306    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-xks-ca" pod "bc1b0625-2cb4-4cae-94bc-95148306ea08" (UID: "bc1b0625-2cb4-4cae-94bc-95148306ea08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:10:36.332996 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.332926    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:36.332996 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.332958    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b0625-2cb4-4cae-94bc-95148306ea08-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:36.332996 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.332973    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfttc\" (UniqueName: \"kubernetes.io/projected/bc1b0625-2cb4-4cae-94bc-95148306ea08-kube-api-access-mfttc\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:36.903308 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.901560    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z" event={"ID":"bc1b0625-2cb4-4cae-94bc-95148306ea08","Type":"ContainerDied","Data":"21f2839340a231d3adbc39017fd9b13191e2931572527061849b2afc1858a61f"}
Nov 03 17:10:36.903308 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.901629    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f2839340a231d3adbc39017fd9b13191e2931572527061849b2afc1858a61f"
Nov 03 17:10:36.903308 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:36.901705    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"
Nov 03 17:10:37.882844 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:10:37.882807    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:10:39.409379 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:39.409350    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:10:39.418485 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:39.418462    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:10:39.538469 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:39.538443    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:10:39.549822 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:39.549798    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:10:40.311632 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:40.311600    5112 scope.go:122] "RemoveContainer" containerID="f39b38f485c4aef09489be0bfab5bd5d56e5b293f3be79a21ed2cb510b09cb48"
Nov 03 17:10:46.059314 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:46.059281    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"]
Nov 03 17:10:46.104849 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:46.104812    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psttb"]
Nov 03 17:10:46.550008 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:46.549969    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36fb9ce-3796-4130-9fdb-3e42f9a56a5b" path="/var/lib/kubelet/pods/f36fb9ce-3796-4130-9fdb-3e42f9a56a5b/volumes"
Nov 03 17:10:51.636308 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.636226    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"]
Nov 03 17:10:51.683745 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.683711    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q6ml9"]
Nov 03 17:10:51.759805 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.759771    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"]
Nov 03 17:10:51.760280 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.760258    5112 cpu_manager.go:414] "containerMap: removing container" podUID="bc1b0625-2cb4-4cae-94bc-95148306ea08" containerName="revoke-keys"
Nov 03 17:10:51.760280 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.760274    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1b0625-2cb4-4cae-94bc-95148306ea08" containerName="revoke-keys"
Nov 03 17:10:51.760374 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.760366    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1b0625-2cb4-4cae-94bc-95148306ea08" containerName="revoke-keys"
Nov 03 17:10:51.763788 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.763771    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.819194 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.816107    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:10:51.819194 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.816501    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:10:51.819194 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.819124    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"]
Nov 03 17:10:51.893969 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.893945    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxv2\" (UniqueName: \"kubernetes.io/projected/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-kube-api-access-zdxv2\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.894139 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.894016    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.894139 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.894085    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.994620 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.994589    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.994805 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.994687    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxv2\" (UniqueName: \"kubernetes.io/projected/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-kube-api-access-zdxv2\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.994805 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.994759    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.995563 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.995543    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:51.997349 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:51.997315    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:52.052149 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:52.052121    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxv2\" (UniqueName: \"kubernetes.io/projected/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-kube-api-access-zdxv2\") pod \"maas-api-revoke-sub-3d4156f1-qslmd\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:52.079159 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:52.079132    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:52.467517 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:52.467488    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"]
Nov 03 17:10:52.468969 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:10:52.468938    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672dbdce_4ad7_4e30_8acc_0b30aeed2ff3.slice/crio-20242fb412ce69c67df489616c4208ce950772f817c4f0982cf09934b2019cc5 WatchSource:0}: Error finding container 20242fb412ce69c67df489616c4208ce950772f817c4f0982cf09934b2019cc5: Status 404 returned error can't find the container with id 20242fb412ce69c67df489616c4208ce950772f817c4f0982cf09934b2019cc5
Nov 03 17:10:52.550788 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:52.550754    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c622aa9-2161-43d8-9378-53eac7081833" path="/var/lib/kubelet/pods/3c622aa9-2161-43d8-9378-53eac7081833/volumes"
Nov 03 17:10:53.003095 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:53.003059    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd" event={"ID":"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3","Type":"ContainerStarted","Data":"04b0e50dc7cde634cd1fdeff80a94008b94962fb0c3d247c7f4ee7b889a4f4f9"}
Nov 03 17:10:53.003523 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:53.003116    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd" event={"ID":"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3","Type":"ContainerStarted","Data":"20242fb412ce69c67df489616c4208ce950772f817c4f0982cf09934b2019cc5"}
Nov 03 17:10:53.117607 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:53.117517    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd" podStartSLOduration=2.117504114 podStartE2EDuration="2.117504114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:53.116798512 +0000 UTC m=+3614.696102367" watchObservedRunningTime="2026-11-03 17:10:53.117504114 +0000 UTC m=+3614.696807989"
Nov 03 17:10:54.010684 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:54.010652    5112 generic.go:356] "Generic (PLEG): container finished" podID="672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" containerID="04b0e50dc7cde634cd1fdeff80a94008b94962fb0c3d247c7f4ee7b889a4f4f9" exitCode=0
Nov 03 17:10:54.011080 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:54.010707    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd" event={"ID":"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3","Type":"ContainerDied","Data":"04b0e50dc7cde634cd1fdeff80a94008b94962fb0c3d247c7f4ee7b889a4f4f9"}
Nov 03 17:10:55.164903 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.164878    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:55.223763 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.223731    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-xks-ca\") pod \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") "
Nov 03 17:10:55.223937 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.223779    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-kube-api-access-zdxv2\" (UniqueName: \"kubernetes.io/projected/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-kube-api-access-zdxv2\") pod \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") "
Nov 03 17:10:55.223937 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.223872    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-openshift-ca\") pod \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\" (UID: \"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3\") "
Nov 03 17:10:55.224280 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.224255    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-openshift-ca" pod "672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" (UID: "672dbdce-4ad7-4e30-8acc-0b30aeed2ff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:10:55.225894 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.225869    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-xks-ca" pod "672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" (UID: "672dbdce-4ad7-4e30-8acc-0b30aeed2ff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:10:55.225989 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.225954    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-kube-api-access-zdxv2" pod "672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" (UID: "672dbdce-4ad7-4e30-8acc-0b30aeed2ff3"). InnerVolumeSpecName "kube-api-access-zdxv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:10:55.325112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.325041    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:55.325112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.325071    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxv2\" (UniqueName: \"kubernetes.io/projected/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-kube-api-access-zdxv2\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:55.325112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:55.325082    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:10:56.023260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:56.023230    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"
Nov 03 17:10:56.023260 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:56.023240    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd" event={"ID":"672dbdce-4ad7-4e30-8acc-0b30aeed2ff3","Type":"ContainerDied","Data":"20242fb412ce69c67df489616c4208ce950772f817c4f0982cf09934b2019cc5"}
Nov 03 17:10:56.023480 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:56.023295    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20242fb412ce69c67df489616c4208ce950772f817c4f0982cf09934b2019cc5"
Nov 03 17:10:59.059556 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:59.059521    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"]
Nov 03 17:10:59.106574 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:10:59.106545    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-cs4cc"]
Nov 03 17:11:00.549214 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:00.549175    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb954bf-4ad5-41f0-ad19-766c9a2b3baa" path="/var/lib/kubelet/pods/2fb954bf-4ad5-41f0-ad19-766c9a2b3baa/volumes"
Nov 03 17:11:07.945856 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:11:07.945820    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:11:37.883137 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:11:37.883087    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:11:40.335612 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:40.335580    5112 scope.go:122] "RemoveContainer" containerID="dfea03295a5575e54ccda936ecd70c718a9aa2eb7b8fe8d53fd8db52c3967826"
Nov 03 17:11:46.059804 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:46.059757    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"]
Nov 03 17:11:46.106722 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:46.106690    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xzd7p"]
Nov 03 17:11:46.548825 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:46.548795    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0" path="/var/lib/kubelet/pods/d27cc0cb-b7a3-46d6-ae43-7948c6a2b9e0/volumes"
Nov 03 17:11:55.044645 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.044613    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"]
Nov 03 17:11:55.045084 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.045072    5112 cpu_manager.go:414] "containerMap: removing container" podUID="672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" containerName="revoke-keys"
Nov 03 17:11:55.045131 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.045085    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" containerName="revoke-keys"
Nov 03 17:11:55.045213 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.045188    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" containerName="revoke-keys"
Nov 03 17:11:55.048508 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.048493    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.146639 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.146590    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 17:11:55.146639 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.146611    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7k42d\""
Nov 03 17:11:55.147038 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.147018    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-433fd6-serving-cert\""
Nov 03 17:11:55.151357 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.151331    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"]
Nov 03 17:11:55.153848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.153829    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jksnq\" (UniqueName: \"kubernetes.io/projected/f9db7043-eca4-4164-ad8b-66881556eade-kube-api-access-jksnq\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.153936 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.153858    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-metrics-certs\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.154021 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.154002    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.154060 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.154034    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9db7043-eca4-4164-ad8b-66881556eade-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.255489 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.255457    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jksnq\" (UniqueName: \"kubernetes.io/projected/f9db7043-eca4-4164-ad8b-66881556eade-kube-api-access-jksnq\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.255691 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.255496    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-metrics-certs\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.255691 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.255583    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.255691 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.255611    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9db7043-eca4-4164-ad8b-66881556eade-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.255872 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:11:55.255686    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-serving-cert: secret "maas-api-e2e-worker-single-main-433fd6-serving-cert" not found
Nov 03 17:11:55.255872 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:11:55.255742    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls podName:f9db7043-eca4-4164-ad8b-66881556eade nodeName:}" failed. No retries permitted until 2026-11-03 17:11:55.755727391 +0000 UTC m=+3677.335031238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls") pod "maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5" (UID: "f9db7043-eca4-4164-ad8b-66881556eade") : secret "maas-api-e2e-worker-single-main-433fd6-serving-cert" not found
Nov 03 17:11:55.256361 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.256343    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9db7043-eca4-4164-ad8b-66881556eade-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.258028 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.258009    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-metrics-certs\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.313911 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.313828    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jksnq\" (UniqueName: \"kubernetes.io/projected/f9db7043-eca4-4164-ad8b-66881556eade-kube-api-access-jksnq\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.761636 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.761598    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.764037 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.764009    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:55.962493 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:55.962456    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:56.339656 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:56.339630    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"]
Nov 03 17:11:56.341078 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:11:56.341055    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9db7043_eca4_4164_ad8b_66881556eade.slice/crio-2d30c787a5c8952380dbb9b88223b40f392dfe1edbd799eee21dae2f00aaec3e WatchSource:0}: Error finding container 2d30c787a5c8952380dbb9b88223b40f392dfe1edbd799eee21dae2f00aaec3e: Status 404 returned error can't find the container with id 2d30c787a5c8952380dbb9b88223b40f392dfe1edbd799eee21dae2f00aaec3e
Nov 03 17:11:56.342845 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:56.342827    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:11:56.427152 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:56.427114    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5" event={"ID":"f9db7043-eca4-4164-ad8b-66881556eade","Type":"ContainerStarted","Data":"2d30c787a5c8952380dbb9b88223b40f392dfe1edbd799eee21dae2f00aaec3e"}
Nov 03 17:11:57.062008 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:57.061970    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"]
Nov 03 17:11:57.108854 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:57.108821    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6kzg"]
Nov 03 17:11:57.434896 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:57.434851    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5" event={"ID":"f9db7043-eca4-4164-ad8b-66881556eade","Type":"ContainerStarted","Data":"163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123"}
Nov 03 17:11:57.435377 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:57.434936    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:11:57.553566 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:57.553515    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5" podStartSLOduration=2.62037447 podStartE2EDuration="3.55350112s" totalImagesPullingTime="933.12665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:11:57.551276831 +0000 UTC m=+3679.130580713" watchObservedRunningTime="2026-11-03 17:11:57.55350112 +0000 UTC m=+3679.132804974"
Nov 03 17:11:58.551159 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:11:58.551114    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9871dc26-5ffe-4076-9061-d8732bda38db" path="/var/lib/kubelet/pods/9871dc26-5ffe-4076-9061-d8732bda38db/volumes"
Nov 03 17:12:03.448874 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:03.448843    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:12:07.883171 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:12:07.883113    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:12:31.058551 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:31.058510    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"]
Nov 03 17:12:31.105536 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:31.105501    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rspzt"]
Nov 03 17:12:32.549389 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:32.549348    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3329d1-f6ed-4628-a553-08ed45ddcffc" path="/var/lib/kubelet/pods/6c3329d1-f6ed-4628-a553-08ed45ddcffc/volumes"
Nov 03 17:12:37.883359 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:12:37.883318    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:12:40.360285 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:40.360247    5112 scope.go:122] "RemoveContainer" containerID="2e9029b81026e74f43148ac3f762c0a6fad7b4816e55fb45a2d9d82ff967cfc5"
Nov 03 17:12:40.369866 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:40.369842    5112 scope.go:122] "RemoveContainer" containerID="e68324453045e37e15b26c8ce5485afcfea195d270a64b75c08e6a7fe45a5f29"
Nov 03 17:12:59.059637 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:59.059600    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"]
Nov 03 17:12:59.106397 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:12:59.106369    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4ch8f"]
Nov 03 17:13:00.548872 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:00.548839    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442dcb04-f937-4606-bed3-9b46b69618f3" path="/var/lib/kubelet/pods/442dcb04-f937-4606-bed3-9b46b69618f3/volumes"
Nov 03 17:13:07.882479 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:13:07.882440    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:13:30.084229 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:30.084182    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"]
Nov 03 17:13:30.131434 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:30.131405    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9wp4"]
Nov 03 17:13:30.548590 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:30.548559    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0597c33-f90c-4c71-b7bc-3cbaa419da26" path="/var/lib/kubelet/pods/d0597c33-f90c-4c71-b7bc-3cbaa419da26/volumes"
Nov 03 17:13:37.882860 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:13:37.882816    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:13:40.015087 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:40.015047    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"]
Nov 03 17:13:40.015648 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:40.015297    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5" podUID="f9db7043-eca4-4164-ad8b-66881556eade" containerName="maas-api" containerID="cri-o://163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123" gracePeriod=30
Nov 03 17:13:40.401372 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:40.401340    5112 scope.go:122] "RemoveContainer" containerID="0f8d3f27e9cf0753d6a423907f083afac5e3edfe93eb8b05d6ae0ac7407fca56"
Nov 03 17:13:40.413697 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:40.412088    5112 scope.go:122] "RemoveContainer" containerID="4ab2170431f10bf3c4efc68be9f2b261c25fc8dd9cf3792a2cd0b5a44d3bc1c0"
Nov 03 17:13:40.423581 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:40.423554    5112 scope.go:122] "RemoveContainer" containerID="fa96d84cff1439919c981a95e1d917a51d3cad95162ccfe03adfedff57b902f8"
Nov 03 17:13:41.281334 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.281299    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:13:41.345961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.345931    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9db7043-eca4-4164-ad8b-66881556eade-kube-api-access-jksnq\" (UniqueName: \"kubernetes.io/projected/f9db7043-eca4-4164-ad8b-66881556eade-kube-api-access-jksnq\") pod \"f9db7043-eca4-4164-ad8b-66881556eade\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") "
Nov 03 17:13:41.346097 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.346016    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9db7043-eca4-4164-ad8b-66881556eade-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9db7043-eca4-4164-ad8b-66881556eade-openshift-service-ca\") pod \"f9db7043-eca4-4164-ad8b-66881556eade\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") "
Nov 03 17:13:41.346097 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.346051    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-metrics-certs\") pod \"f9db7043-eca4-4164-ad8b-66881556eade\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") "
Nov 03 17:13:41.346180 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.346095    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls\") pod \"f9db7043-eca4-4164-ad8b-66881556eade\" (UID: \"f9db7043-eca4-4164-ad8b-66881556eade\") "
Nov 03 17:13:41.346395 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.346371    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9db7043-eca4-4164-ad8b-66881556eade-openshift-service-ca" pod "f9db7043-eca4-4164-ad8b-66881556eade" (UID: "f9db7043-eca4-4164-ad8b-66881556eade"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:13:41.346545 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.346527    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9db7043-eca4-4164-ad8b-66881556eade-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:13:41.348227 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.348146    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-metrics-certs" pod "f9db7043-eca4-4164-ad8b-66881556eade" (UID: "f9db7043-eca4-4164-ad8b-66881556eade"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:13:41.348227 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.348179    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9db7043-eca4-4164-ad8b-66881556eade-kube-api-access-jksnq" pod "f9db7043-eca4-4164-ad8b-66881556eade" (UID: "f9db7043-eca4-4164-ad8b-66881556eade"). InnerVolumeSpecName "kube-api-access-jksnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:13:41.348372 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.348237    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls" pod "f9db7043-eca4-4164-ad8b-66881556eade" (UID: "f9db7043-eca4-4164-ad8b-66881556eade"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:13:41.447895 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.447864    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jksnq\" (UniqueName: \"kubernetes.io/projected/f9db7043-eca4-4164-ad8b-66881556eade-kube-api-access-jksnq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:13:41.447895 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.447891    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:13:41.448104 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:41.447908    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9db7043-eca4-4164-ad8b-66881556eade-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:13:42.093921 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.093881    5112 generic.go:356] "Generic (PLEG): container finished" podID="f9db7043-eca4-4164-ad8b-66881556eade" containerID="163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123" exitCode=0
Nov 03 17:13:42.094132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.093926    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"
Nov 03 17:13:42.094132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.093970    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5" event={"ID":"f9db7043-eca4-4164-ad8b-66881556eade","Type":"ContainerDied","Data":"163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123"}
Nov 03 17:13:42.094132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.094026    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5" event={"ID":"f9db7043-eca4-4164-ad8b-66881556eade","Type":"ContainerDied","Data":"2d30c787a5c8952380dbb9b88223b40f392dfe1edbd799eee21dae2f00aaec3e"}
Nov 03 17:13:42.094132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.094050    5112 scope.go:122] "RemoveContainer" containerID="163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123"
Nov 03 17:13:42.106285 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.106260    5112 scope.go:122] "RemoveContainer" containerID="163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123"
Nov 03 17:13:42.106560 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:13:42.106534    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123\": container with ID starting with 163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123 not found: ID does not exist" containerID="163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123"
Nov 03 17:13:42.106682 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.106570    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123"} err="failed to get container status \"163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123\": rpc error: code = NotFound desc = could not find container \"163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123\": container with ID starting with 163018d825cdd1cf1ca7d190819c6147195d5f7753d6961cf4862bb6630a2123 not found: ID does not exist"
Nov 03 17:13:42.255695 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.255657    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"]
Nov 03 17:13:42.261796 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.261768    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-433fd6-97ff6f48-v9js5"]
Nov 03 17:13:42.548961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:42.548924    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9db7043-eca4-4164-ad8b-66881556eade" path="/var/lib/kubelet/pods/f9db7043-eca4-4164-ad8b-66881556eade/volumes"
Nov 03 17:13:54.060963 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.060882    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"]
Nov 03 17:13:54.108008 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.107977    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nsw6k"]
Nov 03 17:13:54.422802 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.422767    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"]
Nov 03 17:13:54.431075 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.431049    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qslmd"]
Nov 03 17:13:54.549031 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.548995    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672dbdce-4ad7-4e30-8acc-0b30aeed2ff3" path="/var/lib/kubelet/pods/672dbdce-4ad7-4e30-8acc-0b30aeed2ff3/volumes"
Nov 03 17:13:54.549538 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.549521    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf50353-b8f4-40ed-9028-00cc902c155d" path="/var/lib/kubelet/pods/bbf50353-b8f4-40ed-9028-00cc902c155d/volumes"
Nov 03 17:13:54.551164 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.551145    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"]
Nov 03 17:13:54.551622 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.551608    5112 cpu_manager.go:414] "containerMap: removing container" podUID="f9db7043-eca4-4164-ad8b-66881556eade" containerName="maas-api"
Nov 03 17:13:54.551677 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.551623    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9db7043-eca4-4164-ad8b-66881556eade" containerName="maas-api"
Nov 03 17:13:54.551751 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.551721    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9db7043-eca4-4164-ad8b-66881556eade" containerName="maas-api"
Nov 03 17:13:54.556262 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.556243    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.613599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.613562    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:13:54.613599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.613562    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:13:54.616495 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.616465    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"]
Nov 03 17:13:54.684294 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.684181    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.684474 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.684303    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.684474 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.684361    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcpk\" (UniqueName: \"kubernetes.io/projected/055b9883-3fd6-4f01-b030-8391576821dc-kube-api-access-mkcpk\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.785333 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.785297    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.785530 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.785356    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.785530 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.785393    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcpk\" (UniqueName: \"kubernetes.io/projected/055b9883-3fd6-4f01-b030-8391576821dc-kube-api-access-mkcpk\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.785975 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.785951    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.787848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.787823    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.843675 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.843618    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcpk\" (UniqueName: \"kubernetes.io/projected/055b9883-3fd6-4f01-b030-8391576821dc-kube-api-access-mkcpk\") pod \"maas-api-revoke-sub-3d4156f1-qcwxd\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:54.870016 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:54.869984    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:55.240830 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:55.240801    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"]
Nov 03 17:13:55.242666 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:13:55.242622    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055b9883_3fd6_4f01_b030_8391576821dc.slice/crio-65ed1c555c3219c7286c7943bc70675822fd89b6020c12b7bae49e4c2ce93fa2 WatchSource:0}: Error finding container 65ed1c555c3219c7286c7943bc70675822fd89b6020c12b7bae49e4c2ce93fa2: Status 404 returned error can't find the container with id 65ed1c555c3219c7286c7943bc70675822fd89b6020c12b7bae49e4c2ce93fa2
Nov 03 17:13:56.185874 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:56.185841    5112 generic.go:356] "Generic (PLEG): container finished" podID="055b9883-3fd6-4f01-b030-8391576821dc" containerID="a3d97b142f537d9ae0255f5a83547ee9cf2e35831d005399e493e75ad02deb95" exitCode=0
Nov 03 17:13:56.186069 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:56.185893    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd" event={"ID":"055b9883-3fd6-4f01-b030-8391576821dc","Type":"ContainerDied","Data":"a3d97b142f537d9ae0255f5a83547ee9cf2e35831d005399e493e75ad02deb95"}
Nov 03 17:13:56.186069 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:56.185991    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd" event={"ID":"055b9883-3fd6-4f01-b030-8391576821dc","Type":"ContainerStarted","Data":"65ed1c555c3219c7286c7943bc70675822fd89b6020c12b7bae49e4c2ce93fa2"}
Nov 03 17:13:57.328599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.328566    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:57.410034 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.410003    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-openshift-ca\") pod \"055b9883-3fd6-4f01-b030-8391576821dc\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") "
Nov 03 17:13:57.410231 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.410103    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055b9883-3fd6-4f01-b030-8391576821dc-kube-api-access-mkcpk\" (UniqueName: \"kubernetes.io/projected/055b9883-3fd6-4f01-b030-8391576821dc-kube-api-access-mkcpk\") pod \"055b9883-3fd6-4f01-b030-8391576821dc\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") "
Nov 03 17:13:57.410231 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.410185    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-xks-ca\") pod \"055b9883-3fd6-4f01-b030-8391576821dc\" (UID: \"055b9883-3fd6-4f01-b030-8391576821dc\") "
Nov 03 17:13:57.410485 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.410462    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-openshift-ca" pod "055b9883-3fd6-4f01-b030-8391576821dc" (UID: "055b9883-3fd6-4f01-b030-8391576821dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:13:57.410580 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.410564    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:13:57.412352 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.412329    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-xks-ca" pod "055b9883-3fd6-4f01-b030-8391576821dc" (UID: "055b9883-3fd6-4f01-b030-8391576821dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:13:57.412422 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.412403    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055b9883-3fd6-4f01-b030-8391576821dc-kube-api-access-mkcpk" pod "055b9883-3fd6-4f01-b030-8391576821dc" (UID: "055b9883-3fd6-4f01-b030-8391576821dc"). InnerVolumeSpecName "kube-api-access-mkcpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:13:57.512111 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.512038    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkcpk\" (UniqueName: \"kubernetes.io/projected/055b9883-3fd6-4f01-b030-8391576821dc-kube-api-access-mkcpk\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:13:57.512111 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:57.512068    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b9883-3fd6-4f01-b030-8391576821dc-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:13:58.199466 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:58.199434    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"
Nov 03 17:13:58.199656 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:58.199472    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd" event={"ID":"055b9883-3fd6-4f01-b030-8391576821dc","Type":"ContainerDied","Data":"65ed1c555c3219c7286c7943bc70675822fd89b6020c12b7bae49e4c2ce93fa2"}
Nov 03 17:13:58.199656 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:13:58.199531    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ed1c555c3219c7286c7943bc70675822fd89b6020c12b7bae49e4c2ce93fa2"
Nov 03 17:14:07.882940 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:14:07.882906    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:14:29.197434 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.197395    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"]
Nov 03 17:14:29.197934 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.197577    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk" podUID="c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" containerName="manager" containerID="cri-o://fdebe5e9de11e9e206adf5ddf6d277c5c20dcef81608c248b0362d8989b4fe7f" gracePeriod=10
Nov 03 17:14:29.406245 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.406210    5112 generic.go:356] "Generic (PLEG): container finished" podID="c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" containerID="fdebe5e9de11e9e206adf5ddf6d277c5c20dcef81608c248b0362d8989b4fe7f" exitCode=0
Nov 03 17:14:29.406424 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.406234    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk" event={"ID":"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984","Type":"ContainerDied","Data":"fdebe5e9de11e9e206adf5ddf6d277c5c20dcef81608c248b0362d8989b4fe7f"}
Nov 03 17:14:29.458642 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.458585    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 17:14:29.526530 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.526499    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-extensions-socket-volume\") pod \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") "
Nov 03 17:14:29.526674 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.526641    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-kube-api-access-fp4vw\" (UniqueName: \"kubernetes.io/projected/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-kube-api-access-fp4vw\") pod \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\" (UID: \"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984\") "
Nov 03 17:14:29.526861 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.526837    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-extensions-socket-volume" pod "c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" (UID: "c8bb58f6-9b4f-4db3-9c87-3b1deb5de984"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:14:29.526951 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.526934    5112 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-extensions-socket-volume\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:14:29.528676 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.528650    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-kube-api-access-fp4vw" pod "c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" (UID: "c8bb58f6-9b4f-4db3-9c87-3b1deb5de984"). InnerVolumeSpecName "kube-api-access-fp4vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:14:29.627463 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:29.627433    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp4vw\" (UniqueName: \"kubernetes.io/projected/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984-kube-api-access-fp4vw\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:14:30.412912 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:30.412883    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"
Nov 03 17:14:30.413339 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:30.412916    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk" event={"ID":"c8bb58f6-9b4f-4db3-9c87-3b1deb5de984","Type":"ContainerDied","Data":"3fe56ecf921661f28c8ecdc2a20a322adb8f93b22caae3b4113bd08a46f55930"}
Nov 03 17:14:30.413339 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:30.412980    5112 scope.go:122] "RemoveContainer" containerID="fdebe5e9de11e9e206adf5ddf6d277c5c20dcef81608c248b0362d8989b4fe7f"
Nov 03 17:14:30.621865 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:30.621834    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"]
Nov 03 17:14:30.668967 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:30.668906    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdmmk"]
Nov 03 17:14:32.548579 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:32.548548    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" path="/var/lib/kubelet/pods/c8bb58f6-9b4f-4db3-9c87-3b1deb5de984/volumes"
Nov 03 17:14:37.883440 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:14:37.883403    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:14:40.458192 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:40.458149    5112 scope.go:122] "RemoveContainer" containerID="b255a2192b04450443f31cce701ff2b7058362c43b1f031dd6f0670760c0bcc0"
Nov 03 17:14:40.467167 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:14:40.467138    5112 scope.go:122] "RemoveContainer" containerID="998854d363ef7588d320f60b92f57a581bf65f884f0db7d0a562c0c97cddd2b4"
Nov 03 17:15:00.172870 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.172829    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"]
Nov 03 17:15:00.173376 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.173335    5112 cpu_manager.go:414] "containerMap: removing container" podUID="c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" containerName="manager"
Nov 03 17:15:00.173376 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.173348    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" containerName="manager"
Nov 03 17:15:00.173376 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.173356    5112 cpu_manager.go:414] "containerMap: removing container" podUID="055b9883-3fd6-4f01-b030-8391576821dc" containerName="revoke-keys"
Nov 03 17:15:00.173376 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.173362    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055b9883-3fd6-4f01-b030-8391576821dc" containerName="revoke-keys"
Nov 03 17:15:00.173509 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.173443    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bb58f6-9b4f-4db3-9c87-3b1deb5de984" containerName="manager"
Nov 03 17:15:00.173509 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.173450    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="055b9883-3fd6-4f01-b030-8391576821dc" containerName="revoke-keys"
Nov 03 17:15:00.176618 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.176595    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.218887 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.218852    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da79d06d-3f66-41f0-a01c-91a71cda1a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895435-8g25p\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.219071 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.218922    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nmq\" (UniqueName: \"kubernetes.io/projected/da79d06d-3f66-41f0-a01c-91a71cda1a15-kube-api-access-k8nmq\") pod \"maas-api-key-cleanup-29895435-8g25p\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.232490 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.232459    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"]
Nov 03 17:15:00.282720 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.282686    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:15:00.319634 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.319593    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da79d06d-3f66-41f0-a01c-91a71cda1a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895435-8g25p\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.319816 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.319681    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nmq\" (UniqueName: \"kubernetes.io/projected/da79d06d-3f66-41f0-a01c-91a71cda1a15-kube-api-access-k8nmq\") pod \"maas-api-key-cleanup-29895435-8g25p\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.320378 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.320355    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da79d06d-3f66-41f0-a01c-91a71cda1a15-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895435-8g25p\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.376678 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.376642    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nmq\" (UniqueName: \"kubernetes.io/projected/da79d06d-3f66-41f0-a01c-91a71cda1a15-kube-api-access-k8nmq\") pod \"maas-api-key-cleanup-29895435-8g25p\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.491581 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.491483    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:00.862596 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:00.862568    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"]
Nov 03 17:15:00.864992 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:15:00.864965    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda79d06d_3f66_41f0_a01c_91a71cda1a15.slice/crio-b4882532b03f0b8b0439b13d7161e784a8d4c6ed2b4a412f9e7784863041a7ee WatchSource:0}: Error finding container b4882532b03f0b8b0439b13d7161e784a8d4c6ed2b4a412f9e7784863041a7ee: Status 404 returned error can't find the container with id b4882532b03f0b8b0439b13d7161e784a8d4c6ed2b4a412f9e7784863041a7ee
Nov 03 17:15:01.611889 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:01.611805    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" event={"ID":"da79d06d-3f66-41f0-a01c-91a71cda1a15","Type":"ContainerStarted","Data":"79acbd28e1a219e5b543e0af69941b072bba7af72769300c9a9e375a0f7b3774"}
Nov 03 17:15:01.611889 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:01.611856    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" event={"ID":"da79d06d-3f66-41f0-a01c-91a71cda1a15","Type":"ContainerStarted","Data":"b4882532b03f0b8b0439b13d7161e784a8d4c6ed2b4a412f9e7784863041a7ee"}
Nov 03 17:15:01.723208 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:01.723160    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" podStartSLOduration=1.7231458530000001 podStartE2EDuration="1.723145853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:01.722488503 +0000 UTC m=+3863.301792354" watchObservedRunningTime="2026-11-03 17:15:01.723145853 +0000 UTC m=+3863.302449708"
Nov 03 17:15:02.619956 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:02.619920    5112 generic.go:356] "Generic (PLEG): container finished" podID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerID="79acbd28e1a219e5b543e0af69941b072bba7af72769300c9a9e375a0f7b3774" exitCode=60
Nov 03 17:15:02.620484 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:02.620002    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" event={"ID":"da79d06d-3f66-41f0-a01c-91a71cda1a15","Type":"ContainerDied","Data":"79acbd28e1a219e5b543e0af69941b072bba7af72769300c9a9e375a0f7b3774"}
Nov 03 17:15:02.620484 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:02.620287    5112 scope.go:122] "RemoveContainer" containerID="79acbd28e1a219e5b543e0af69941b072bba7af72769300c9a9e375a0f7b3774"
Nov 03 17:15:03.628387 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:03.628287    5112 generic.go:356] "Generic (PLEG): container finished" podID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerID="f7bf70347ab654635af50e53fbf1add050a8d42b6a08630b42214596e2d98bda" exitCode=60
Nov 03 17:15:03.628387 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:03.628322    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" event={"ID":"da79d06d-3f66-41f0-a01c-91a71cda1a15","Type":"ContainerDied","Data":"f7bf70347ab654635af50e53fbf1add050a8d42b6a08630b42214596e2d98bda"}
Nov 03 17:15:03.628915 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:03.628433    5112 scope.go:122] "RemoveContainer" containerID="79acbd28e1a219e5b543e0af69941b072bba7af72769300c9a9e375a0f7b3774"
Nov 03 17:15:03.628915 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:03.628631    5112 scope.go:122] "RemoveContainer" containerID="f7bf70347ab654635af50e53fbf1add050a8d42b6a08630b42214596e2d98bda"
Nov 03 17:15:03.629005 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:15:03.628924    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895435-8g25p_odh-ai-gateway-infra(da79d06d-3f66-41f0-a01c-91a71cda1a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15"
Nov 03 17:15:04.635659 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:04.635632    5112 scope.go:122] "RemoveContainer" containerID="f7bf70347ab654635af50e53fbf1add050a8d42b6a08630b42214596e2d98bda"
Nov 03 17:15:04.636050 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:15:04.635861    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895435-8g25p_odh-ai-gateway-infra(da79d06d-3f66-41f0-a01c-91a71cda1a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15"
Nov 03 17:15:07.883268 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:15:07.883232    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:15:16.542368 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:16.542339    5112 scope.go:122] "RemoveContainer" containerID="f7bf70347ab654635af50e53fbf1add050a8d42b6a08630b42214596e2d98bda"
Nov 03 17:15:17.707307 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:17.707272    5112 generic.go:356] "Generic (PLEG): container finished" podID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerID="701ee6afab50313299f1d09a1c1db77abc0f3b278a0ffa27085d5dcdf05bd471" exitCode=60
Nov 03 17:15:17.707709 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:17.707344    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" event={"ID":"da79d06d-3f66-41f0-a01c-91a71cda1a15","Type":"ContainerDied","Data":"701ee6afab50313299f1d09a1c1db77abc0f3b278a0ffa27085d5dcdf05bd471"}
Nov 03 17:15:17.707709 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:17.707402    5112 scope.go:122] "RemoveContainer" containerID="f7bf70347ab654635af50e53fbf1add050a8d42b6a08630b42214596e2d98bda"
Nov 03 17:15:17.707828 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:17.707753    5112 scope.go:122] "RemoveContainer" containerID="701ee6afab50313299f1d09a1c1db77abc0f3b278a0ffa27085d5dcdf05bd471"
Nov 03 17:15:17.708042 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:15:17.708015    5112 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895435-8g25p_odh-ai-gateway-infra(da79d06d-3f66-41f0-a01c-91a71cda1a15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15"
Nov 03 17:15:18.832455 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:18.832421    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"]
Nov 03 17:15:18.982652 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:18.982628    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:19.114831 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.114752    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da79d06d-3f66-41f0-a01c-91a71cda1a15-kube-api-access-k8nmq\" (UniqueName: \"kubernetes.io/projected/da79d06d-3f66-41f0-a01c-91a71cda1a15-kube-api-access-k8nmq\") pod \"da79d06d-3f66-41f0-a01c-91a71cda1a15\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") "
Nov 03 17:15:19.114983 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.114946    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da79d06d-3f66-41f0-a01c-91a71cda1a15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da79d06d-3f66-41f0-a01c-91a71cda1a15-maas-api-service-ca\") pod \"da79d06d-3f66-41f0-a01c-91a71cda1a15\" (UID: \"da79d06d-3f66-41f0-a01c-91a71cda1a15\") "
Nov 03 17:15:19.115409 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.115377    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da79d06d-3f66-41f0-a01c-91a71cda1a15-maas-api-service-ca" pod "da79d06d-3f66-41f0-a01c-91a71cda1a15" (UID: "da79d06d-3f66-41f0-a01c-91a71cda1a15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:15:19.116955 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.116932    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da79d06d-3f66-41f0-a01c-91a71cda1a15-kube-api-access-k8nmq" pod "da79d06d-3f66-41f0-a01c-91a71cda1a15" (UID: "da79d06d-3f66-41f0-a01c-91a71cda1a15"). InnerVolumeSpecName "kube-api-access-k8nmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:15:19.216466 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.216431    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8nmq\" (UniqueName: \"kubernetes.io/projected/da79d06d-3f66-41f0-a01c-91a71cda1a15-kube-api-access-k8nmq\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:15:19.216466 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.216464    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da79d06d-3f66-41f0-a01c-91a71cda1a15-maas-api-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:15:19.721550 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.721521    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"
Nov 03 17:15:19.721550 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.721534    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p" event={"ID":"da79d06d-3f66-41f0-a01c-91a71cda1a15","Type":"ContainerDied","Data":"b4882532b03f0b8b0439b13d7161e784a8d4c6ed2b4a412f9e7784863041a7ee"}
Nov 03 17:15:19.721814 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.721602    5112 scope.go:122] "RemoveContainer" containerID="701ee6afab50313299f1d09a1c1db77abc0f3b278a0ffa27085d5dcdf05bd471"
Nov 03 17:15:19.884411 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.884363    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"]
Nov 03 17:15:19.889719 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:19.889694    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-8g25p"]
Nov 03 17:15:20.549483 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:20.549447    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" path="/var/lib/kubelet/pods/da79d06d-3f66-41f0-a01c-91a71cda1a15/volumes"
Nov 03 17:15:26.057691 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:26.057652    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"]
Nov 03 17:15:26.104907 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:26.104870    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-h8fvq"]
Nov 03 17:15:26.549156 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:26.549118    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d6cb8c-0346-4df7-92e2-b2c2507d5801" path="/var/lib/kubelet/pods/f1d6cb8c-0346-4df7-92e2-b2c2507d5801/volumes"
Nov 03 17:15:37.058556 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:37.058519    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"]
Nov 03 17:15:37.105322 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:37.105284    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dtc8z"]
Nov 03 17:15:37.883814 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:15:37.883771    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:15:38.548981 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:38.548939    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1b0625-2cb4-4cae-94bc-95148306ea08" path="/var/lib/kubelet/pods/bc1b0625-2cb4-4cae-94bc-95148306ea08/volumes"
Nov 03 17:15:39.449562 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:39.449535    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:15:39.459891 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:39.459864    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:15:39.576806 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:39.576781    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:15:39.586923 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:39.586903    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:15:40.515001 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:40.514970    5112 scope.go:122] "RemoveContainer" containerID="54dd6ab260368386aefeea36d626d65a97d563413c1c521cde0471f2e310ada6"
Nov 03 17:15:40.523969 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:40.523948    5112 scope.go:122] "RemoveContainer" containerID="08ec9c96deb9f3f46fce8505495755b41384928f40160502802736a8233f549e"
Nov 03 17:15:40.532762 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:40.532743    5112 scope.go:122] "RemoveContainer" containerID="0f266a333c5787b5fde9983ee0f2f0fe1adfcc420349a0ef842aa82f0c9791e7"
Nov 03 17:15:42.813617 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.813580    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"]
Nov 03 17:15:42.814066 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814051    5112 cpu_manager.go:414] "containerMap: removing container" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814067    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814079    5112 cpu_manager.go:414] "containerMap: removing container" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814085    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814093    5112 cpu_manager.go:414] "containerMap: removing container" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814098    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814335 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814172    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814335 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814181    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.814335 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.814188    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="da79d06d-3f66-41f0-a01c-91a71cda1a15" containerName="cleanup"
Nov 03 17:15:42.819190 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.819168    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:42.850369 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.850342    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e16d16c1-4f21-4007-9dbf-c66a4fd290a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-thzcr\" (UID: \"e16d16c1-4f21-4007-9dbf-c66a4fd290a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:42.850480 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.850407    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxtb\" (UniqueName: \"kubernetes.io/projected/e16d16c1-4f21-4007-9dbf-c66a4fd290a0-kube-api-access-2jxtb\") pod \"kuadrant-operator-controller-manager-678d87766f-thzcr\" (UID: \"e16d16c1-4f21-4007-9dbf-c66a4fd290a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:42.868907 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.868881    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"]
Nov 03 17:15:42.912540 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.912516    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bmzc9\""
Nov 03 17:15:42.951143 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.951112    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e16d16c1-4f21-4007-9dbf-c66a4fd290a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-thzcr\" (UID: \"e16d16c1-4f21-4007-9dbf-c66a4fd290a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:42.951304 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.951177    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxtb\" (UniqueName: \"kubernetes.io/projected/e16d16c1-4f21-4007-9dbf-c66a4fd290a0-kube-api-access-2jxtb\") pod \"kuadrant-operator-controller-manager-678d87766f-thzcr\" (UID: \"e16d16c1-4f21-4007-9dbf-c66a4fd290a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:42.951586 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:42.951566    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e16d16c1-4f21-4007-9dbf-c66a4fd290a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-thzcr\" (UID: \"e16d16c1-4f21-4007-9dbf-c66a4fd290a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:43.008647 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:43.008620    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxtb\" (UniqueName: \"kubernetes.io/projected/e16d16c1-4f21-4007-9dbf-c66a4fd290a0-kube-api-access-2jxtb\") pod \"kuadrant-operator-controller-manager-678d87766f-thzcr\" (UID: \"e16d16c1-4f21-4007-9dbf-c66a4fd290a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:43.132447 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:43.132381    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:43.602838 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:15:43.602811    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16d16c1_4f21_4007_9dbf_c66a4fd290a0.slice/crio-1b07f6bfc0da3cef29515f491ecbb462d711a48fe9e950b9ef116a580fc1615f WatchSource:0}: Error finding container 1b07f6bfc0da3cef29515f491ecbb462d711a48fe9e950b9ef116a580fc1615f: Status 404 returned error can't find the container with id 1b07f6bfc0da3cef29515f491ecbb462d711a48fe9e950b9ef116a580fc1615f
Nov 03 17:15:43.647838 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:43.647812    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"]
Nov 03 17:15:43.857876 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:43.857786    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr" event={"ID":"e16d16c1-4f21-4007-9dbf-c66a4fd290a0","Type":"ContainerStarted","Data":"d50c2401d8e7ceb21f01ddea56ac2efcb07419d136acd55914d35803e4991ea3"}
Nov 03 17:15:43.857876 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:43.857848    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr" event={"ID":"e16d16c1-4f21-4007-9dbf-c66a4fd290a0","Type":"ContainerStarted","Data":"1b07f6bfc0da3cef29515f491ecbb462d711a48fe9e950b9ef116a580fc1615f"}
Nov 03 17:15:43.858329 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:43.857878    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:15:43.985315 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:43.985262    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr" podStartSLOduration=1.985246859 podStartE2EDuration="1.985246859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:43.984173283 +0000 UTC m=+3905.563477138" watchObservedRunningTime="2026-11-03 17:15:43.985246859 +0000 UTC m=+3905.564550916"
Nov 03 17:15:54.866687 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:15:54.866653    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-thzcr"
Nov 03 17:16:07.462298 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.462261    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"]
Nov 03 17:16:07.466406 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.466380    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.523858 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.523824    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"]
Nov 03 17:16:07.531927 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.531901    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:16:07.532072 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.531978    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:16:07.543641 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.543613    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.543796 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.543690    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blz7h\" (UniqueName: \"kubernetes.io/projected/566ce642-1b28-4c00-a53e-d76933ec159a-kube-api-access-blz7h\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.543843 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.543811    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.644789 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.644746    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.644993 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.644827    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blz7h\" (UniqueName: \"kubernetes.io/projected/566ce642-1b28-4c00-a53e-d76933ec159a-kube-api-access-blz7h\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.644993 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.644864    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.645633 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.645609    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.647317 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.647285    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.702096 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.702059    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blz7h\" (UniqueName: \"kubernetes.io/projected/566ce642-1b28-4c00-a53e-d76933ec159a-kube-api-access-blz7h\") pod \"maas-api-revoke-sub-d576e68b-tcx5c\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.781112 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:07.781024    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:07.885150 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:16:07.885095    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:16:08.156560 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:08.156530    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"]
Nov 03 17:16:08.158247 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:16:08.158222    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566ce642_1b28_4c00_a53e_d76933ec159a.slice/crio-682af820bf6f92ba403add5f0369bf01dd39c0133e7af0deb010e82bd60b0735 WatchSource:0}: Error finding container 682af820bf6f92ba403add5f0369bf01dd39c0133e7af0deb010e82bd60b0735: Status 404 returned error can't find the container with id 682af820bf6f92ba403add5f0369bf01dd39c0133e7af0deb010e82bd60b0735
Nov 03 17:16:09.041793 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:09.041761    5112 generic.go:356] "Generic (PLEG): container finished" podID="566ce642-1b28-4c00-a53e-d76933ec159a" containerID="84b4e92691232fa4703b9690485383226962b4633d404b2ee78134d81d9e6c4a" exitCode=0
Nov 03 17:16:09.042153 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:09.041793    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c" event={"ID":"566ce642-1b28-4c00-a53e-d76933ec159a","Type":"ContainerDied","Data":"84b4e92691232fa4703b9690485383226962b4633d404b2ee78134d81d9e6c4a"}
Nov 03 17:16:09.042153 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:09.041901    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c" event={"ID":"566ce642-1b28-4c00-a53e-d76933ec159a","Type":"ContainerStarted","Data":"682af820bf6f92ba403add5f0369bf01dd39c0133e7af0deb010e82bd60b0735"}
Nov 03 17:16:10.204924 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.204894    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:10.274578 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.274540    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566ce642-1b28-4c00-a53e-d76933ec159a-kube-api-access-blz7h\" (UniqueName: \"kubernetes.io/projected/566ce642-1b28-4c00-a53e-d76933ec159a-kube-api-access-blz7h\") pod \"566ce642-1b28-4c00-a53e-d76933ec159a\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") "
Nov 03 17:16:10.274744 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.274668    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-xks-ca\") pod \"566ce642-1b28-4c00-a53e-d76933ec159a\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") "
Nov 03 17:16:10.274744 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.274716    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-openshift-ca\") pod \"566ce642-1b28-4c00-a53e-d76933ec159a\" (UID: \"566ce642-1b28-4c00-a53e-d76933ec159a\") "
Nov 03 17:16:10.275271 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.275243    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-openshift-ca" pod "566ce642-1b28-4c00-a53e-d76933ec159a" (UID: "566ce642-1b28-4c00-a53e-d76933ec159a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:16:10.276832 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.276794    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566ce642-1b28-4c00-a53e-d76933ec159a-kube-api-access-blz7h" pod "566ce642-1b28-4c00-a53e-d76933ec159a" (UID: "566ce642-1b28-4c00-a53e-d76933ec159a"). InnerVolumeSpecName "kube-api-access-blz7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:16:10.276930 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.276913    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-xks-ca" pod "566ce642-1b28-4c00-a53e-d76933ec159a" (UID: "566ce642-1b28-4c00-a53e-d76933ec159a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:16:10.375865 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.375771    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:16:10.375865 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.375811    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566ce642-1b28-4c00-a53e-d76933ec159a-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:16:10.375865 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:10.375827    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blz7h\" (UniqueName: \"kubernetes.io/projected/566ce642-1b28-4c00-a53e-d76933ec159a-kube-api-access-blz7h\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:16:11.054237 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:11.054188    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"
Nov 03 17:16:11.054237 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:11.054228    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c" event={"ID":"566ce642-1b28-4c00-a53e-d76933ec159a","Type":"ContainerDied","Data":"682af820bf6f92ba403add5f0369bf01dd39c0133e7af0deb010e82bd60b0735"}
Nov 03 17:16:11.054451 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:11.054278    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682af820bf6f92ba403add5f0369bf01dd39c0133e7af0deb010e82bd60b0735"
Nov 03 17:16:36.592827 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.592792    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"]
Nov 03 17:16:36.593291 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.593276    5112 cpu_manager.go:414] "containerMap: removing container" podUID="566ce642-1b28-4c00-a53e-d76933ec159a" containerName="revoke-keys"
Nov 03 17:16:36.593349 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.593291    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566ce642-1b28-4c00-a53e-d76933ec159a" containerName="revoke-keys"
Nov 03 17:16:36.593401 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.593390    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="566ce642-1b28-4c00-a53e-d76933ec159a" containerName="revoke-keys"
Nov 03 17:16:36.596745 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.596726    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.648484 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.648457    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:16:36.666096 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.666051    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.666327 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.666299    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.666424 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.666410    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rj6\" (UniqueName: \"kubernetes.io/projected/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-kube-api-access-f6rj6\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.694530 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.694497    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:16:36.697050 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.697022    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"]
Nov 03 17:16:36.767966 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.767929    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.768171 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.768021    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.768171 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.768056    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rj6\" (UniqueName: \"kubernetes.io/projected/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-kube-api-access-f6rj6\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.768668 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.768642    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.770534 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.770509    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.824526 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.824490    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rj6\" (UniqueName: \"kubernetes.io/projected/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-kube-api-access-f6rj6\") pod \"maas-api-revoke-sub-d2bc93bc-2m9ls\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:36.910181 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:36.910149    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:37.280458 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:37.280426    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"]
Nov 03 17:16:37.282534 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:16:37.282504    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c364a8d_dfea_461d_9ba9_3ad505bc1a9b.slice/crio-d9e47e955ce78771679e0d08ec6bed456016233f0ba26edf36b5c718a119199c WatchSource:0}: Error finding container d9e47e955ce78771679e0d08ec6bed456016233f0ba26edf36b5c718a119199c: Status 404 returned error can't find the container with id d9e47e955ce78771679e0d08ec6bed456016233f0ba26edf36b5c718a119199c
Nov 03 17:16:37.881827 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:16:37.881736    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:16:38.224987 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:38.224951    5112 generic.go:356] "Generic (PLEG): container finished" podID="9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" containerID="3c991dca2fc86bf9199ae9f65db9f3ecbcfa5e7ab530b315fe9a3c2d3aa19f95" exitCode=0
Nov 03 17:16:38.225166 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:38.225001    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls" event={"ID":"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b","Type":"ContainerDied","Data":"3c991dca2fc86bf9199ae9f65db9f3ecbcfa5e7ab530b315fe9a3c2d3aa19f95"}
Nov 03 17:16:38.225166 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:38.225063    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls" event={"ID":"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b","Type":"ContainerStarted","Data":"d9e47e955ce78771679e0d08ec6bed456016233f0ba26edf36b5c718a119199c"}
Nov 03 17:16:39.369249 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.369224    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:39.493056 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.492982    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-xks-ca\") pod \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") "
Nov 03 17:16:39.493056 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.493032    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-openshift-ca\") pod \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") "
Nov 03 17:16:39.493302 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.493119    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-kube-api-access-f6rj6\" (UniqueName: \"kubernetes.io/projected/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-kube-api-access-f6rj6\") pod \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\" (UID: \"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b\") "
Nov 03 17:16:39.493497 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.493471    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-openshift-ca" pod "9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" (UID: "9c364a8d-dfea-461d-9ba9-3ad505bc1a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:16:39.495066 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.495036    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-xks-ca" pod "9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" (UID: "9c364a8d-dfea-461d-9ba9-3ad505bc1a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:16:39.495276 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.495252    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-kube-api-access-f6rj6" pod "9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" (UID: "9c364a8d-dfea-461d-9ba9-3ad505bc1a9b"). InnerVolumeSpecName "kube-api-access-f6rj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:16:39.594659 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.594628    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rj6\" (UniqueName: \"kubernetes.io/projected/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-kube-api-access-f6rj6\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:16:39.594659 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.594657    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:16:39.594807 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:39.594667    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:16:40.237246 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:40.237192    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls" event={"ID":"9c364a8d-dfea-461d-9ba9-3ad505bc1a9b","Type":"ContainerDied","Data":"d9e47e955ce78771679e0d08ec6bed456016233f0ba26edf36b5c718a119199c"}
Nov 03 17:16:40.237246 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:40.237239    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"
Nov 03 17:16:40.237246 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:40.237254    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e47e955ce78771679e0d08ec6bed456016233f0ba26edf36b5c718a119199c"
Nov 03 17:16:40.574140 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:40.574053    5112 scope.go:122] "RemoveContainer" containerID="def1449210cb9aa612a1012bde47f2a97a5550560a174bf67c3d9bfcb1897cc5"
Nov 03 17:16:40.583451 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:16:40.583431    5112 scope.go:122] "RemoveContainer" containerID="7be1757afae265cfe67c44966fc9eb10720d33522dac2d55b22bf56f1d8ffb72"
Nov 03 17:17:00.343648 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.343569    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"]
Nov 03 17:17:00.344285 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.344265    5112 cpu_manager.go:414] "containerMap: removing container" podUID="9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" containerName="revoke-keys"
Nov 03 17:17:00.344507 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.344286    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" containerName="revoke-keys"
Nov 03 17:17:00.344507 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.344465    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" containerName="revoke-keys"
Nov 03 17:17:00.347872 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.347851    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.393737 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.393711    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.393839 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.393753    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a18070-c328-427c-a713-6f36e80cd93d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.393898 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.393836    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.393898 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.393885    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxb44\" (UniqueName: \"kubernetes.io/projected/b5a18070-c328-427c-a713-6f36e80cd93d-kube-api-access-bxb44\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.448869 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.448838    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a56144-serving-cert\""
Nov 03 17:17:00.449098 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.448889    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7k42d\""
Nov 03 17:17:00.449098 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.449006    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 17:17:00.451465 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.451443    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"]
Nov 03 17:17:00.494675 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.494641    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxb44\" (UniqueName: \"kubernetes.io/projected/b5a18070-c328-427c-a713-6f36e80cd93d-kube-api-access-bxb44\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.494848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.494763    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.494848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.494800    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a18070-c328-427c-a713-6f36e80cd93d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.494848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.494837    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.495358 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:17:00.495050    5112 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-serving-cert: secret "maas-api-e2e-worker-single-main-a56144-serving-cert" not found
Nov 03 17:17:00.495358 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:17:00.495133    5112 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls podName:b5a18070-c328-427c-a713-6f36e80cd93d nodeName:}" failed. No retries permitted until 2026-11-03 17:17:00.995111939 +0000 UTC m=+3982.574415789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls") pod "maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" (UID: "b5a18070-c328-427c-a713-6f36e80cd93d") : secret "maas-api-e2e-worker-single-main-a56144-serving-cert" not found
Nov 03 17:17:00.495862 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.495831    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a18070-c328-427c-a713-6f36e80cd93d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.497457 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.497434    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.553804 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.553774    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxb44\" (UniqueName: \"kubernetes.io/projected/b5a18070-c328-427c-a713-6f36e80cd93d-kube-api-access-bxb44\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:00.999316 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:00.999273    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:01.001694 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:01.001669    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:01.260922 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:01.260840    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:01.633310 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:01.633277    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"]
Nov 03 17:17:01.634727 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:17:01.634698    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a18070_c328_427c_a713_6f36e80cd93d.slice/crio-6fb0cf7a80200710d3e4ba23b2f2cdda6583f78d76f9783ce5006305c1fd1302 WatchSource:0}: Error finding container 6fb0cf7a80200710d3e4ba23b2f2cdda6583f78d76f9783ce5006305c1fd1302: Status 404 returned error can't find the container with id 6fb0cf7a80200710d3e4ba23b2f2cdda6583f78d76f9783ce5006305c1fd1302
Nov 03 17:17:01.636175 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:01.636159    5112 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:17:02.366381 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:02.366345    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" event={"ID":"b5a18070-c328-427c-a713-6f36e80cd93d","Type":"ContainerStarted","Data":"6fb0cf7a80200710d3e4ba23b2f2cdda6583f78d76f9783ce5006305c1fd1302"}
Nov 03 17:17:03.374001 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:03.373963    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" event={"ID":"b5a18070-c328-427c-a713-6f36e80cd93d","Type":"ContainerStarted","Data":"c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828"}
Nov 03 17:17:03.374482 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:03.374056    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:03.492433 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:03.492376    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" podStartSLOduration=2.643587423 podStartE2EDuration="3.492362987s" totalImagesPullingTime="848.775564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:03.490433612 +0000 UTC m=+3985.069737466" watchObservedRunningTime="2026-11-03 17:17:03.492362987 +0000 UTC m=+3985.071666875"
Nov 03 17:17:07.882432 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:17:07.882393    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:17:09.385827 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:09.385796    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:17:37.883573 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:17:37.883532    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:17:40.610550 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:17:40.610520    5112 scope.go:122] "RemoveContainer" containerID="04b0e50dc7cde634cd1fdeff80a94008b94962fb0c3d247c7f4ee7b889a4f4f9"
Nov 03 17:18:07.882006 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:18:07.881961    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:18:24.060042 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:24.059958    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"]
Nov 03 17:18:24.060609 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:24.060194    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" podUID="b5a18070-c328-427c-a713-6f36e80cd93d" containerName="maas-api" containerID="cri-o://c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828" gracePeriod=30
Nov 03 17:18:24.380545 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:24.380447    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" podUID="b5a18070-c328-427c-a713-6f36e80cd93d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.170:8443/health\": dial tcp 10.132.0.170:8443: connect: connection refused"
Nov 03 17:18:25.327750 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.327724    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:18:25.355768 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.355728    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls\") pod \"b5a18070-c328-427c-a713-6f36e80cd93d\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") "
Nov 03 17:18:25.355938 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.355822    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-metrics-certs\") pod \"b5a18070-c328-427c-a713-6f36e80cd93d\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") "
Nov 03 17:18:25.356416 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.356064    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a18070-c328-427c-a713-6f36e80cd93d-kube-api-access-bxb44\" (UniqueName: \"kubernetes.io/projected/b5a18070-c328-427c-a713-6f36e80cd93d-kube-api-access-bxb44\") pod \"b5a18070-c328-427c-a713-6f36e80cd93d\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") "
Nov 03 17:18:25.357079 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.356657    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a18070-c328-427c-a713-6f36e80cd93d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a18070-c328-427c-a713-6f36e80cd93d-openshift-service-ca\") pod \"b5a18070-c328-427c-a713-6f36e80cd93d\" (UID: \"b5a18070-c328-427c-a713-6f36e80cd93d\") "
Nov 03 17:18:25.357232 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.357083    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a18070-c328-427c-a713-6f36e80cd93d-openshift-service-ca" pod "b5a18070-c328-427c-a713-6f36e80cd93d" (UID: "b5a18070-c328-427c-a713-6f36e80cd93d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:18:25.358146 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.358119    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls" pod "b5a18070-c328-427c-a713-6f36e80cd93d" (UID: "b5a18070-c328-427c-a713-6f36e80cd93d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:25.358410 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.358369    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-metrics-certs" pod "b5a18070-c328-427c-a713-6f36e80cd93d" (UID: "b5a18070-c328-427c-a713-6f36e80cd93d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:18:25.358410 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.358385    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a18070-c328-427c-a713-6f36e80cd93d-kube-api-access-bxb44" pod "b5a18070-c328-427c-a713-6f36e80cd93d" (UID: "b5a18070-c328-427c-a713-6f36e80cd93d"). InnerVolumeSpecName "kube-api-access-bxb44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:18:25.457989 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.457956    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxb44\" (UniqueName: \"kubernetes.io/projected/b5a18070-c328-427c-a713-6f36e80cd93d-kube-api-access-bxb44\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:25.457989 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.457985    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a18070-c328-427c-a713-6f36e80cd93d-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:25.457989 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.457996    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:25.458236 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.458005    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a18070-c328-427c-a713-6f36e80cd93d-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:18:25.862771 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.862724    5112 generic.go:356] "Generic (PLEG): container finished" podID="b5a18070-c328-427c-a713-6f36e80cd93d" containerID="c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828" exitCode=0
Nov 03 17:18:25.862771 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.862775    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" event={"ID":"b5a18070-c328-427c-a713-6f36e80cd93d","Type":"ContainerDied","Data":"c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828"}
Nov 03 17:18:25.863038 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.862817    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9" event={"ID":"b5a18070-c328-427c-a713-6f36e80cd93d","Type":"ContainerDied","Data":"6fb0cf7a80200710d3e4ba23b2f2cdda6583f78d76f9783ce5006305c1fd1302"}
Nov 03 17:18:25.863038 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.862844    5112 scope.go:122] "RemoveContainer" containerID="c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828"
Nov 03 17:18:25.863038 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.862986    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"
Nov 03 17:18:25.873711 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.873690    5112 scope.go:122] "RemoveContainer" containerID="c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828"
Nov 03 17:18:25.873979 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:18:25.873962    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828\": container with ID starting with c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828 not found: ID does not exist" containerID="c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828"
Nov 03 17:18:25.874030 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:25.873986    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828"} err="failed to get container status \"c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828\": rpc error: code = NotFound desc = could not find container \"c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828\": container with ID starting with c17329368496c7ca2026fc7ef731c749e4fd3147a456e671e7597ff793780828 not found: ID does not exist"
Nov 03 17:18:26.022251 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:26.022221    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"]
Nov 03 17:18:26.025749 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:26.025721    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a56144-6c67dcbb65-q6kk9"]
Nov 03 17:18:26.548866 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:26.548792    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a18070-c328-427c-a713-6f36e80cd93d" path="/var/lib/kubelet/pods/b5a18070-c328-427c-a713-6f36e80cd93d/volumes"
Nov 03 17:18:37.883579 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:18:37.883539    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:18:39.869687 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:39.869651    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"]
Nov 03 17:18:39.870193 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:39.870179    5112 cpu_manager.go:414] "containerMap: removing container" podUID="b5a18070-c328-427c-a713-6f36e80cd93d" containerName="maas-api"
Nov 03 17:18:39.870193 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:39.870194    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a18070-c328-427c-a713-6f36e80cd93d" containerName="maas-api"
Nov 03 17:18:39.870302 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:39.870279    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a18070-c328-427c-a713-6f36e80cd93d" containerName="maas-api"
Nov 03 17:18:39.874944 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:39.874923    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:39.972394 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:39.972362    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"]
Nov 03 17:18:39.973927 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:39.973906    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-dd9d4320-kserve-self-signed-certs\""
Nov 03 17:18:40.007477 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.007440    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tls-certs\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.007683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.007485    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-model-cache\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.007683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.007511    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-home\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.007683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.007657    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-dshm\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.007859 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.007800    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tmp-dir\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.007859 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.007835    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxh4\" (UniqueName: \"kubernetes.io/projected/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-kube-api-access-chxh4\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.108699 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.108658    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tmp-dir\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.108699 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.108702    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxh4\" (UniqueName: \"kubernetes.io/projected/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-kube-api-access-chxh4\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.108961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.108744    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tls-certs\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.108961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.108770    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-model-cache\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.108961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.108789    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-home\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.108961 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.108823    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-dshm\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.109174 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.109143    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tmp-dir\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.109264 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.109210    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-model-cache\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.109333 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.109315    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-home\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.111151 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.111133    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-dshm\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.111469 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.111447    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tls-certs\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.165465 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.165428    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxh4\" (UniqueName: \"kubernetes.io/projected/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-kube-api-access-chxh4\") pod \"e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") " pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.187237 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.187181    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:40.571156 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.571128    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"]
Nov 03 17:18:40.572604 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:18:40.572567    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5f0aa51_9c65_443f_8fe2_a4b1798860ff.slice/crio-538e67afd98f2a39c61064793392c12109deee0b888e99ac4033a5c9dffb160c WatchSource:0}: Error finding container 538e67afd98f2a39c61064793392c12109deee0b888e99ac4033a5c9dffb160c: Status 404 returned error can't find the container with id 538e67afd98f2a39c61064793392c12109deee0b888e99ac4033a5c9dffb160c
Nov 03 17:18:40.956911 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.956871    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4" event={"ID":"e5f0aa51-9c65-443f-8fe2-a4b1798860ff","Type":"ContainerStarted","Data":"ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d"}
Nov 03 17:18:40.956911 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:40.956915    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4" event={"ID":"e5f0aa51-9c65-443f-8fe2-a4b1798860ff","Type":"ContainerStarted","Data":"538e67afd98f2a39c61064793392c12109deee0b888e99ac4033a5c9dffb160c"}
Nov 03 17:18:41.071529 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:41.071469    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4" podStartSLOduration=2.071455912 podStartE2EDuration="2.071455912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:18:41.069003773 +0000 UTC m=+4082.648307649" watchObservedRunningTime="2026-11-03 17:18:41.071455912 +0000 UTC m=+4082.650759766"
Nov 03 17:18:41.187548 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:41.187510    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:47.850354 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:47.850312    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2jzfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:18:47.850985 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:47.850375    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2jzfr" podUID="c132a838-d7ba-4445-a3e7-f5e4955d4688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:18:50.188093 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:50.188060    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:50.202820 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:50.202789    5112 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:51.063519 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:51.063483    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:18:58.057627 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:58.057592    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"]
Nov 03 17:18:58.104502 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:58.104464    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qcwxd"]
Nov 03 17:18:58.550133 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:18:58.550101    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055b9883-3fd6-4f01-b030-8391576821dc" path="/var/lib/kubelet/pods/055b9883-3fd6-4f01-b030-8391576821dc/volumes"
Nov 03 17:19:07.882437 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:19:07.882397    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:19:31.441374 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:31.441335    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:19:31.443839 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:31.441827    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:19:37.882574 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:19:37.882535    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:19:57.228369 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.228286    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"]
Nov 03 17:19:57.232464 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.232442    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.284224 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.284190    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:19:57.330371 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.330338    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:19:57.334512 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.334485    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"]
Nov 03 17:19:57.396279 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.396249    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.396449 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.396290    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.396449 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.396318    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kjg\" (UniqueName: \"kubernetes.io/projected/bba87545-0ffb-4a00-be73-a8d764f83385-kube-api-access-k5kjg\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.497449 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.497374    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.497449 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.497422    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.497683 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.497452    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kjg\" (UniqueName: \"kubernetes.io/projected/bba87545-0ffb-4a00-be73-a8d764f83385-kube-api-access-k5kjg\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.498116 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.498093    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.499818 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.499794    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.554042 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.554015    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kjg\" (UniqueName: \"kubernetes.io/projected/bba87545-0ffb-4a00-be73-a8d764f83385-kube-api-access-k5kjg\") pod \"maas-api-revoke-sub-452b6d7b-8jc7t\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:57.846691 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:57.846609    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:19:58.230852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:58.230825    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"]
Nov 03 17:19:58.231587 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:19:58.231554    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba87545_0ffb_4a00_be73_a8d764f83385.slice/crio-10695dc8ee25c55b07a29f13306057c40c89b9b406d762c169cba69b22ec5df6 WatchSource:0}: Error finding container 10695dc8ee25c55b07a29f13306057c40c89b9b406d762c169cba69b22ec5df6: Status 404 returned error can't find the container with id 10695dc8ee25c55b07a29f13306057c40c89b9b406d762c169cba69b22ec5df6
Nov 03 17:19:58.462900 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:58.462864    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t" event={"ID":"bba87545-0ffb-4a00-be73-a8d764f83385","Type":"ContainerStarted","Data":"10695dc8ee25c55b07a29f13306057c40c89b9b406d762c169cba69b22ec5df6"}
Nov 03 17:19:59.472763 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:59.472728    5112 generic.go:356] "Generic (PLEG): container finished" podID="bba87545-0ffb-4a00-be73-a8d764f83385" containerID="15ad08322cba56b7172d406a62572403ccd5ae6d5a88a50a2367cfee0eeed39f" exitCode=0
Nov 03 17:19:59.473334 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:19:59.472820    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t" event={"ID":"bba87545-0ffb-4a00-be73-a8d764f83385","Type":"ContainerDied","Data":"15ad08322cba56b7172d406a62572403ccd5ae6d5a88a50a2367cfee0eeed39f"}
Nov 03 17:20:01.053938 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.053908    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:20:01.114221 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.113504    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba87545-0ffb-4a00-be73-a8d764f83385-kube-api-access-k5kjg\" (UniqueName: \"kubernetes.io/projected/bba87545-0ffb-4a00-be73-a8d764f83385-kube-api-access-k5kjg\") pod \"bba87545-0ffb-4a00-be73-a8d764f83385\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") "
Nov 03 17:20:01.114221 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.113687    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-xks-ca\") pod \"bba87545-0ffb-4a00-be73-a8d764f83385\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") "
Nov 03 17:20:01.114221 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.113780    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-openshift-ca\") pod \"bba87545-0ffb-4a00-be73-a8d764f83385\" (UID: \"bba87545-0ffb-4a00-be73-a8d764f83385\") "
Nov 03 17:20:01.115078 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.115049    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-openshift-ca" pod "bba87545-0ffb-4a00-be73-a8d764f83385" (UID: "bba87545-0ffb-4a00-be73-a8d764f83385"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:20:01.122509 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.122481    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-xks-ca" pod "bba87545-0ffb-4a00-be73-a8d764f83385" (UID: "bba87545-0ffb-4a00-be73-a8d764f83385"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:20:01.124266 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.124238    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba87545-0ffb-4a00-be73-a8d764f83385-kube-api-access-k5kjg" pod "bba87545-0ffb-4a00-be73-a8d764f83385" (UID: "bba87545-0ffb-4a00-be73-a8d764f83385"). InnerVolumeSpecName "kube-api-access-k5kjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:20:01.238394 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.238301    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:01.238630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.238608    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba87545-0ffb-4a00-be73-a8d764f83385-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:01.238747 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.238733    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kjg\" (UniqueName: \"kubernetes.io/projected/bba87545-0ffb-4a00-be73-a8d764f83385-kube-api-access-k5kjg\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:01.451337 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.451300    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:20:01.451552 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.451370    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:20:01.528379 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.528293    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"
Nov 03 17:20:01.528681 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.528331    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t" event={"ID":"bba87545-0ffb-4a00-be73-a8d764f83385","Type":"ContainerDied","Data":"10695dc8ee25c55b07a29f13306057c40c89b9b406d762c169cba69b22ec5df6"}
Nov 03 17:20:01.528787 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:01.528722    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10695dc8ee25c55b07a29f13306057c40c89b9b406d762c169cba69b22ec5df6"
Nov 03 17:20:07.883254 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:20:07.883213    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:20:10.175526 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:10.175484    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"]
Nov 03 17:20:16.450350 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:16.450313    5112 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-2wnwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:20:16.450791 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:16.450382    5112 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-2wnwx" podUID="106b94ab-7184-4c6b-b0e4-3bf6a4b763a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:20:17.653871 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.653833    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"]
Nov 03 17:20:17.654423 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.654406    5112 cpu_manager.go:414] "containerMap: removing container" podUID="bba87545-0ffb-4a00-be73-a8d764f83385" containerName="revoke-keys"
Nov 03 17:20:17.654473 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.654425    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba87545-0ffb-4a00-be73-a8d764f83385" containerName="revoke-keys"
Nov 03 17:20:17.654543 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.654533    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba87545-0ffb-4a00-be73-a8d764f83385" containerName="revoke-keys"
Nov 03 17:20:17.658021 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.657989    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.714380 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.714344    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"]
Nov 03 17:20:17.746852 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.746814    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swskm\" (UniqueName: \"kubernetes.io/projected/a65d54f3-ba19-47e4-bd17-c84b911335f7-kube-api-access-swskm\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.747039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.746950    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.747039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.747013    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.757691 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.757659    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:20:17.757838 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.757702    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:20:17.848148 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.848096    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swskm\" (UniqueName: \"kubernetes.io/projected/a65d54f3-ba19-47e4-bd17-c84b911335f7-kube-api-access-swskm\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.848399 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.848183    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.848399 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.848335    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.849040 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.849013    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.850677 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.850660    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.909706 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.909613    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swskm\" (UniqueName: \"kubernetes.io/projected/a65d54f3-ba19-47e4-bd17-c84b911335f7-kube-api-access-swskm\") pod \"maas-api-revoke-sub-3e31a506-flkm6\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:17.971535 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:17.971497    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:18.345660 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:18.345625    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"]
Nov 03 17:20:18.347432 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:20:18.347406    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65d54f3_ba19_47e4_bd17_c84b911335f7.slice/crio-f350dd0c08d20eb9ddf32c098f8070e3244635907fd13a726d72e5068e839b20 WatchSource:0}: Error finding container f350dd0c08d20eb9ddf32c098f8070e3244635907fd13a726d72e5068e839b20: Status 404 returned error can't find the container with id f350dd0c08d20eb9ddf32c098f8070e3244635907fd13a726d72e5068e839b20
Nov 03 17:20:18.653971 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:18.653940    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6" event={"ID":"a65d54f3-ba19-47e4-bd17-c84b911335f7","Type":"ContainerStarted","Data":"f350dd0c08d20eb9ddf32c098f8070e3244635907fd13a726d72e5068e839b20"}
Nov 03 17:20:19.662210 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:19.662159    5112 generic.go:356] "Generic (PLEG): container finished" podID="a65d54f3-ba19-47e4-bd17-c84b911335f7" containerID="db7c5084c7f877185a068b853644e82f49fdbb66aa3acbf384a1a82f6f21a779" exitCode=0
Nov 03 17:20:19.662646 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:19.662246    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6" event={"ID":"a65d54f3-ba19-47e4-bd17-c84b911335f7","Type":"ContainerDied","Data":"db7c5084c7f877185a068b853644e82f49fdbb66aa3acbf384a1a82f6f21a779"}
Nov 03 17:20:20.840832 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:20.840795    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:20.982573 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:20.982474    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-openshift-ca\") pod \"a65d54f3-ba19-47e4-bd17-c84b911335f7\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") "
Nov 03 17:20:20.982744 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:20.982613    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-xks-ca\") pod \"a65d54f3-ba19-47e4-bd17-c84b911335f7\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") "
Nov 03 17:20:20.982744 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:20.982651    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65d54f3-ba19-47e4-bd17-c84b911335f7-kube-api-access-swskm\" (UniqueName: \"kubernetes.io/projected/a65d54f3-ba19-47e4-bd17-c84b911335f7-kube-api-access-swskm\") pod \"a65d54f3-ba19-47e4-bd17-c84b911335f7\" (UID: \"a65d54f3-ba19-47e4-bd17-c84b911335f7\") "
Nov 03 17:20:20.984040 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:20.983993    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-openshift-ca" pod "a65d54f3-ba19-47e4-bd17-c84b911335f7" (UID: "a65d54f3-ba19-47e4-bd17-c84b911335f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:20:20.987941 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:20.986561    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-xks-ca" pod "a65d54f3-ba19-47e4-bd17-c84b911335f7" (UID: "a65d54f3-ba19-47e4-bd17-c84b911335f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:20:20.987941 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:20.986697    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65d54f3-ba19-47e4-bd17-c84b911335f7-kube-api-access-swskm" pod "a65d54f3-ba19-47e4-bd17-c84b911335f7" (UID: "a65d54f3-ba19-47e4-bd17-c84b911335f7"). InnerVolumeSpecName "kube-api-access-swskm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:20:21.084010 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:21.083969    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swskm\" (UniqueName: \"kubernetes.io/projected/a65d54f3-ba19-47e4-bd17-c84b911335f7-kube-api-access-swskm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:21.084010 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:21.084013    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:21.084285 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:21.084031    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a65d54f3-ba19-47e4-bd17-c84b911335f7-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:21.696222 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:21.694628    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"
Nov 03 17:20:21.700224 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:21.699342    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6" event={"ID":"a65d54f3-ba19-47e4-bd17-c84b911335f7","Type":"ContainerDied","Data":"f350dd0c08d20eb9ddf32c098f8070e3244635907fd13a726d72e5068e839b20"}
Nov 03 17:20:21.700224 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:21.699406    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f350dd0c08d20eb9ddf32c098f8070e3244635907fd13a726d72e5068e839b20"
Nov 03 17:20:25.196338 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.196264    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4" podUID="e5f0aa51-9c65-443f-8fe2-a4b1798860ff" containerName="main" containerID="cri-o://ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d" gracePeriod=45
Nov 03 17:20:25.452537 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.452474    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:20:25.455875 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.455848    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-kube-api-access-chxh4\" (UniqueName: \"kubernetes.io/projected/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-kube-api-access-chxh4\") pod \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") "
Nov 03 17:20:25.456039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.455923    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-home\") pod \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") "
Nov 03 17:20:25.456039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.455957    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-model-cache\") pod \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") "
Nov 03 17:20:25.456039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456033    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tls-certs\") pod \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") "
Nov 03 17:20:25.456293 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456076    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tmp-dir\") pod \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") "
Nov 03 17:20:25.456293 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456122    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-dshm\") pod \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\" (UID: \"e5f0aa51-9c65-443f-8fe2-a4b1798860ff\") "
Nov 03 17:20:25.456293 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456245    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-home" pod "e5f0aa51-9c65-443f-8fe2-a4b1798860ff" (UID: "e5f0aa51-9c65-443f-8fe2-a4b1798860ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:20:25.456293 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456273    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-model-cache" pod "e5f0aa51-9c65-443f-8fe2-a4b1798860ff" (UID: "e5f0aa51-9c65-443f-8fe2-a4b1798860ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:20:25.456525 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456502    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tmp-dir" pod "e5f0aa51-9c65-443f-8fe2-a4b1798860ff" (UID: "e5f0aa51-9c65-443f-8fe2-a4b1798860ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:20:25.456586 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456561    5112 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-home\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:25.456644 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.456586    5112 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-model-cache\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:25.458175 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.458149    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-kube-api-access-chxh4" pod "e5f0aa51-9c65-443f-8fe2-a4b1798860ff" (UID: "e5f0aa51-9c65-443f-8fe2-a4b1798860ff"). InnerVolumeSpecName "kube-api-access-chxh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:20:25.458680 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.458645    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tls-certs" pod "e5f0aa51-9c65-443f-8fe2-a4b1798860ff" (UID: "e5f0aa51-9c65-443f-8fe2-a4b1798860ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:20:25.459597 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.459565    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-dshm" pod "e5f0aa51-9c65-443f-8fe2-a4b1798860ff" (UID: "e5f0aa51-9c65-443f-8fe2-a4b1798860ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:20:25.557474 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.557300    5112 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tls-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:25.557474 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.557336    5112 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-tmp-dir\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:25.557474 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.557352    5112 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-dshm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:25.557474 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.557366    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxh4\" (UniqueName: \"kubernetes.io/projected/e5f0aa51-9c65-443f-8fe2-a4b1798860ff-kube-api-access-chxh4\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:25.749314 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.749193    5112 generic.go:356] "Generic (PLEG): container finished" podID="e5f0aa51-9c65-443f-8fe2-a4b1798860ff" containerID="ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d" exitCode=0
Nov 03 17:20:25.749314 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.749245    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"
Nov 03 17:20:25.749314 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.749290    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4" event={"ID":"e5f0aa51-9c65-443f-8fe2-a4b1798860ff","Type":"ContainerDied","Data":"ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d"}
Nov 03 17:20:25.749566 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.749333    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4" event={"ID":"e5f0aa51-9c65-443f-8fe2-a4b1798860ff","Type":"ContainerDied","Data":"538e67afd98f2a39c61064793392c12109deee0b888e99ac4033a5c9dffb160c"}
Nov 03 17:20:25.749566 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.749353    5112 scope.go:122] "RemoveContainer" containerID="ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d"
Nov 03 17:20:25.759532 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.759510    5112 scope.go:122] "RemoveContainer" containerID="ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d"
Nov 03 17:20:25.759827 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:20:25.759807    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d\": container with ID starting with ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d not found: ID does not exist" containerID="ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d"
Nov 03 17:20:25.759901 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.759841    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d"} err="failed to get container status \"ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d\": rpc error: code = NotFound desc = could not find container \"ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d\": container with ID starting with ace68ee92d345c10937ce3aa5c62946b8ec3d744585295cccc00fad6c359778d not found: ID does not exist"
Nov 03 17:20:25.910703 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.910663    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"]
Nov 03 17:20:25.917564 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:25.917532    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-dd9d4320-kserve-68f4467d7c-qnql4"]
Nov 03 17:20:26.548680 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:26.548600    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f0aa51-9c65-443f-8fe2-a4b1798860ff" path="/var/lib/kubelet/pods/e5f0aa51-9c65-443f-8fe2-a4b1798860ff/volumes"
Nov 03 17:20:37.883279 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:20:37.883239    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:20:39.489848 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:39.489814    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:20:39.499901 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:39.499873    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:20:39.617631 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:39.617604    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:20:39.627838 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:39.627808    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:20:40.654934 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:40.654901    5112 scope.go:122] "RemoveContainer" containerID="a3d97b142f537d9ae0255f5a83547ee9cf2e35831d005399e493e75ad02deb95"
Nov 03 17:20:55.954025 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.953987    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"]
Nov 03 17:20:55.954664 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.954645    5112 cpu_manager.go:414] "containerMap: removing container" podUID="a65d54f3-ba19-47e4-bd17-c84b911335f7" containerName="revoke-keys"
Nov 03 17:20:55.954718 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.954665    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65d54f3-ba19-47e4-bd17-c84b911335f7" containerName="revoke-keys"
Nov 03 17:20:55.954718 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.954702    5112 cpu_manager.go:414] "containerMap: removing container" podUID="e5f0aa51-9c65-443f-8fe2-a4b1798860ff" containerName="main"
Nov 03 17:20:55.954718 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.954707    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f0aa51-9c65-443f-8fe2-a4b1798860ff" containerName="main"
Nov 03 17:20:55.954818 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.954785    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f0aa51-9c65-443f-8fe2-a4b1798860ff" containerName="main"
Nov 03 17:20:55.954818 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.954796    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65d54f3-ba19-47e4-bd17-c84b911335f7" containerName="revoke-keys"
Nov 03 17:20:55.958037 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:55.958020    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.060039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.060009    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:20:56.060266 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.060052    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fnmxp\""
Nov 03 17:20:56.062910 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.062885    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"]
Nov 03 17:20:56.104262 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.104230    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.104392 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.104328    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/01f73755-c032-44b2-bb80-baa61eaa2ce9-kube-api-access-sm6dh\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.104392 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.104381    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.205332 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.205266    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/01f73755-c032-44b2-bb80-baa61eaa2ce9-kube-api-access-sm6dh\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.205332 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.205315    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.205487 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.205384    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.205977 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.205955    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.207728 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.207700    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.264578 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.264546    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/01f73755-c032-44b2-bb80-baa61eaa2ce9-kube-api-access-sm6dh\") pod \"maas-api-revoke-sub-ffe287a6-rrdvg\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.271333 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.271315    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:56.648735 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.648709    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"]
Nov 03 17:20:56.650496 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:20:56.650463    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f73755_c032_44b2_bb80_baa61eaa2ce9.slice/crio-08c3e600ea67316ca12ed8bc800b0161ca964f45f36057b93d3a9839508488d4 WatchSource:0}: Error finding container 08c3e600ea67316ca12ed8bc800b0161ca964f45f36057b93d3a9839508488d4: Status 404 returned error can't find the container with id 08c3e600ea67316ca12ed8bc800b0161ca964f45f36057b93d3a9839508488d4
Nov 03 17:20:56.996800 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.996770    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg" event={"ID":"01f73755-c032-44b2-bb80-baa61eaa2ce9","Type":"ContainerStarted","Data":"3f46244399841323cb8ee128c1401e8659aa29ab85430a0a06370811207021d9"}
Nov 03 17:20:56.997132 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:56.996831    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg" event={"ID":"01f73755-c032-44b2-bb80-baa61eaa2ce9","Type":"ContainerStarted","Data":"08c3e600ea67316ca12ed8bc800b0161ca964f45f36057b93d3a9839508488d4"}
Nov 03 17:20:57.111084 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:57.111029    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg" podStartSLOduration=2.111014992 podStartE2EDuration="2.111014992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:20:57.108447579 +0000 UTC m=+4218.687751444" watchObservedRunningTime="2026-11-03 17:20:57.111014992 +0000 UTC m=+4218.690318887"
Nov 03 17:20:58.003189 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:58.003151    5112 generic.go:356] "Generic (PLEG): container finished" podID="01f73755-c032-44b2-bb80-baa61eaa2ce9" containerID="3f46244399841323cb8ee128c1401e8659aa29ab85430a0a06370811207021d9" exitCode=0
Nov 03 17:20:58.003709 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:58.003253    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg" event={"ID":"01f73755-c032-44b2-bb80-baa61eaa2ce9","Type":"ContainerDied","Data":"3f46244399841323cb8ee128c1401e8659aa29ab85430a0a06370811207021d9"}
Nov 03 17:20:59.168195 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.168168    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:20:59.338039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.337956    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-openshift-ca\") pod \"01f73755-c032-44b2-bb80-baa61eaa2ce9\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") "
Nov 03 17:20:59.338039 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.338001    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f73755-c032-44b2-bb80-baa61eaa2ce9-kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/01f73755-c032-44b2-bb80-baa61eaa2ce9-kube-api-access-sm6dh\") pod \"01f73755-c032-44b2-bb80-baa61eaa2ce9\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") "
Nov 03 17:20:59.338270 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.338102    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-xks-ca\") pod \"01f73755-c032-44b2-bb80-baa61eaa2ce9\" (UID: \"01f73755-c032-44b2-bb80-baa61eaa2ce9\") "
Nov 03 17:20:59.338446 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.338419    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-openshift-ca" pod "01f73755-c032-44b2-bb80-baa61eaa2ce9" (UID: "01f73755-c032-44b2-bb80-baa61eaa2ce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:20:59.340361 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.340334    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-xks-ca" pod "01f73755-c032-44b2-bb80-baa61eaa2ce9" (UID: "01f73755-c032-44b2-bb80-baa61eaa2ce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:20:59.340474 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.340363    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f73755-c032-44b2-bb80-baa61eaa2ce9-kube-api-access-sm6dh" pod "01f73755-c032-44b2-bb80-baa61eaa2ce9" (UID: "01f73755-c032-44b2-bb80-baa61eaa2ce9"). InnerVolumeSpecName "kube-api-access-sm6dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:20:59.439140 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.439109    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-xks-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:59.439140 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.439136    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f73755-c032-44b2-bb80-baa61eaa2ce9-maas-api-openshift-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:20:59.439140 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:20:59.439149    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6dh\" (UniqueName: \"kubernetes.io/projected/01f73755-c032-44b2-bb80-baa61eaa2ce9-kube-api-access-sm6dh\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:00.015736 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:00.015707    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"
Nov 03 17:21:00.015912 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:00.015711    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg" event={"ID":"01f73755-c032-44b2-bb80-baa61eaa2ce9","Type":"ContainerDied","Data":"08c3e600ea67316ca12ed8bc800b0161ca964f45f36057b93d3a9839508488d4"}
Nov 03 17:21:00.015912 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:00.015866    5112 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c3e600ea67316ca12ed8bc800b0161ca964f45f36057b93d3a9839508488d4"
Nov 03 17:21:07.882364 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:21:07.882321    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:21:09.313362 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.313323    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"]
Nov 03 17:21:09.314044 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.313804    5112 cpu_manager.go:414] "containerMap: removing container" podUID="01f73755-c032-44b2-bb80-baa61eaa2ce9" containerName="revoke-keys"
Nov 03 17:21:09.314044 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.313816    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f73755-c032-44b2-bb80-baa61eaa2ce9" containerName="revoke-keys"
Nov 03 17:21:09.314044 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.313936    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f73755-c032-44b2-bb80-baa61eaa2ce9" containerName="revoke-keys"
Nov 03 17:21:09.317147 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.317122    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.337133 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.337097    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8f24db-ec68-4061-9a03-c32e8c79b046-openshift-service-ca\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.337354 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.337149    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntptm\" (UniqueName: \"kubernetes.io/projected/eb8f24db-ec68-4061-9a03-c32e8c79b046-kube-api-access-ntptm\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.337427 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.337342    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-metrics-certs\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.337427 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.337390    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-maas-api-tls\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.417415 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.417377    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 17:21:09.417670 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.417424    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 17:21:09.417670 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.417482    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7k42d\""
Nov 03 17:21:09.420153 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.420121    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"]
Nov 03 17:21:09.438646 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.438609    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8f24db-ec68-4061-9a03-c32e8c79b046-openshift-service-ca\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.438901 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.438676    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntptm\" (UniqueName: \"kubernetes.io/projected/eb8f24db-ec68-4061-9a03-c32e8c79b046-kube-api-access-ntptm\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.438901 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.438734    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-metrics-certs\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.438901 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.438769    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-maas-api-tls\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.439497 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.439439    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8f24db-ec68-4061-9a03-c32e8c79b046-openshift-service-ca\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.441537 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.441511    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-maas-api-tls\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.441798 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.441780    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-metrics-certs\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.495489 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.495454    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntptm\" (UniqueName: \"kubernetes.io/projected/eb8f24db-ec68-4061-9a03-c32e8c79b046-kube-api-access-ntptm\") pod \"maas-api-dfc545664-2qkb7\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") " pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:09.632524 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:09.632427    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:10.010251 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:10.010224    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"]
Nov 03 17:21:10.011769 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:21:10.011735    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8f24db_ec68_4061_9a03_c32e8c79b046.slice/crio-10fe56807210c408a678ebb7bb82faf1ef6f3592c5731757cc8374becdf62ad8 WatchSource:0}: Error finding container 10fe56807210c408a678ebb7bb82faf1ef6f3592c5731757cc8374becdf62ad8: Status 404 returned error can't find the container with id 10fe56807210c408a678ebb7bb82faf1ef6f3592c5731757cc8374becdf62ad8
Nov 03 17:21:10.080881 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:10.080845    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7" event={"ID":"eb8f24db-ec68-4061-9a03-c32e8c79b046","Type":"ContainerStarted","Data":"10fe56807210c408a678ebb7bb82faf1ef6f3592c5731757cc8374becdf62ad8"}
Nov 03 17:21:11.066676 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:11.066636    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"]
Nov 03 17:21:11.097821 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:11.097725    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7" event={"ID":"eb8f24db-ec68-4061-9a03-c32e8c79b046","Type":"ContainerStarted","Data":"544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7"}
Nov 03 17:21:11.097978 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:11.097952    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:11.113322 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:11.113287    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcx5c"]
Nov 03 17:21:11.213637 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:11.213574    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7" podStartSLOduration=1.405403668 podStartE2EDuration="2.213557731s" totalImagesPullingTime="808.154063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:11.21164447 +0000 UTC m=+4232.790948325" watchObservedRunningTime="2026-11-03 17:21:11.213557731 +0000 UTC m=+4232.792861586"
Nov 03 17:21:12.549363 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:12.549327    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566ce642-1b28-4c00-a53e-d76933ec159a" path="/var/lib/kubelet/pods/566ce642-1b28-4c00-a53e-d76933ec159a/volumes"
Nov 03 17:21:17.109595 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.109563    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:17.462398 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.462353    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"]
Nov 03 17:21:17.468020 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.467985    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.520748 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.520706    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wh94\" (UniqueName: \"kubernetes.io/projected/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-kube-api-access-4wh94\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.520952 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.520782    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-maas-api-tls\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.520952 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.520842    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-metrics-certs\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.520952 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.520912    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-openshift-service-ca\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.522099 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.522066    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"]
Nov 03 17:21:17.621987 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.621947    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wh94\" (UniqueName: \"kubernetes.io/projected/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-kube-api-access-4wh94\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.622229 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.622005    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-maas-api-tls\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.622229 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.622040    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-metrics-certs\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.622229 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.622079    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-openshift-service-ca\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.622819 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.622797    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-openshift-service-ca\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.624609 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.624585    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-metrics-certs\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.624703 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.624645    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-maas-api-tls\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.680787 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.680742    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wh94\" (UniqueName: \"kubernetes.io/projected/1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d-kube-api-access-4wh94\") pod \"maas-api-855b85cfb4-bglrh\" (UID: \"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d\") " pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:17.786047 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:17.785951    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:18.168545 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:18.168514    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"]
Nov 03 17:21:18.169944 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:21:18.169905    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae9dc0f_861f_4546_a2b0_f0ffaf1de33d.slice/crio-18d8e5e5e4e9d54e76bdf064784309fbd370be61cf22c4f7ec717613b6eab958 WatchSource:0}: Error finding container 18d8e5e5e4e9d54e76bdf064784309fbd370be61cf22c4f7ec717613b6eab958: Status 404 returned error can't find the container with id 18d8e5e5e4e9d54e76bdf064784309fbd370be61cf22c4f7ec717613b6eab958
Nov 03 17:21:19.145073 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:19.145031    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh" event={"ID":"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d","Type":"ContainerStarted","Data":"fa0c88ba06f42963f0cb0b6293a000886982989df6e96d70de1d8225f8b825b3"}
Nov 03 17:21:19.145269 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:19.145104    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:19.145269 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:19.145127    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh" event={"ID":"1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d","Type":"ContainerStarted","Data":"18d8e5e5e4e9d54e76bdf064784309fbd370be61cf22c4f7ec717613b6eab958"}
Nov 03 17:21:19.266104 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:19.265999    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh" podStartSLOduration=1.445678559 podStartE2EDuration="2.265984761s" totalImagesPullingTime="820.306202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:19.263854848 +0000 UTC m=+4240.843158715" watchObservedRunningTime="2026-11-03 17:21:19.265984761 +0000 UTC m=+4240.845288616"
Nov 03 17:21:25.157478 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.157444    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-855b85cfb4-bglrh"
Nov 03 17:21:25.330622 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.330586    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"]
Nov 03 17:21:25.330861 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.330816    5112 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7" podUID="eb8f24db-ec68-4061-9a03-c32e8c79b046" containerName="maas-api" containerID="cri-o://544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7" gracePeriod=30
Nov 03 17:21:25.613633 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.613607    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:25.703522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.703438    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-maas-api-tls\") pod \"eb8f24db-ec68-4061-9a03-c32e8c79b046\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") "
Nov 03 17:21:25.703522 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.703511    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-metrics-certs\") pod \"eb8f24db-ec68-4061-9a03-c32e8c79b046\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") "
Nov 03 17:21:25.703719 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.703581    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8f24db-ec68-4061-9a03-c32e8c79b046-kube-api-access-ntptm\" (UniqueName: \"kubernetes.io/projected/eb8f24db-ec68-4061-9a03-c32e8c79b046-kube-api-access-ntptm\") pod \"eb8f24db-ec68-4061-9a03-c32e8c79b046\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") "
Nov 03 17:21:25.703719 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.703658    5112 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8f24db-ec68-4061-9a03-c32e8c79b046-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8f24db-ec68-4061-9a03-c32e8c79b046-openshift-service-ca\") pod \"eb8f24db-ec68-4061-9a03-c32e8c79b046\" (UID: \"eb8f24db-ec68-4061-9a03-c32e8c79b046\") "
Nov 03 17:21:25.704093 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.704065    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8f24db-ec68-4061-9a03-c32e8c79b046-openshift-service-ca" pod "eb8f24db-ec68-4061-9a03-c32e8c79b046" (UID: "eb8f24db-ec68-4061-9a03-c32e8c79b046"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:21:25.705835 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.705812    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-maas-api-tls" pod "eb8f24db-ec68-4061-9a03-c32e8c79b046" (UID: "eb8f24db-ec68-4061-9a03-c32e8c79b046"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:21:25.705928 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.705816    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8f24db-ec68-4061-9a03-c32e8c79b046-kube-api-access-ntptm" pod "eb8f24db-ec68-4061-9a03-c32e8c79b046" (UID: "eb8f24db-ec68-4061-9a03-c32e8c79b046"). InnerVolumeSpecName "kube-api-access-ntptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:21:25.705928 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.705898    5112 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-metrics-certs" pod "eb8f24db-ec68-4061-9a03-c32e8c79b046" (UID: "eb8f24db-ec68-4061-9a03-c32e8c79b046"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:21:25.804693 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.804652    5112 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8f24db-ec68-4061-9a03-c32e8c79b046-openshift-service-ca\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:25.804693 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.804687    5112 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-maas-api-tls\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:25.804693 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.804700    5112 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8f24db-ec68-4061-9a03-c32e8c79b046-metrics-certs\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:25.804943 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:25.804708    5112 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntptm\" (UniqueName: \"kubernetes.io/projected/eb8f24db-ec68-4061-9a03-c32e8c79b046-kube-api-access-ntptm\") on node \"ip-10-0-137-122.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 17:21:26.187438 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.187401    5112 generic.go:356] "Generic (PLEG): container finished" podID="eb8f24db-ec68-4061-9a03-c32e8c79b046" containerID="544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7" exitCode=0
Nov 03 17:21:26.187862 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.187454    5112 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"
Nov 03 17:21:26.187862 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.187496    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7" event={"ID":"eb8f24db-ec68-4061-9a03-c32e8c79b046","Type":"ContainerDied","Data":"544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7"}
Nov 03 17:21:26.187862 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.187552    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dfc545664-2qkb7" event={"ID":"eb8f24db-ec68-4061-9a03-c32e8c79b046","Type":"ContainerDied","Data":"10fe56807210c408a678ebb7bb82faf1ef6f3592c5731757cc8374becdf62ad8"}
Nov 03 17:21:26.187862 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.187582    5112 scope.go:122] "RemoveContainer" containerID="544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7"
Nov 03 17:21:26.198164 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.198146    5112 scope.go:122] "RemoveContainer" containerID="544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7"
Nov 03 17:21:26.198545 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:21:26.198523    5112 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7\": container with ID starting with 544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7 not found: ID does not exist" containerID="544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7"
Nov 03 17:21:26.198628 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.198552    5112 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7"} err="failed to get container status \"544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7\": rpc error: code = NotFound desc = could not find container \"544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7\": container with ID starting with 544e152388c7639820419734c393e5b25ad12d2781498e0e4bb81ab4638904a7 not found: ID does not exist"
Nov 03 17:21:26.355008 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.354977    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"]
Nov 03 17:21:26.402133 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.402046    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dfc545664-2qkb7"]
Nov 03 17:21:26.550028 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:26.549990    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8f24db-ec68-4061-9a03-c32e8c79b046" path="/var/lib/kubelet/pods/eb8f24db-ec68-4061-9a03-c32e8c79b046/volumes"
Nov 03 17:21:37.883162 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:21:37.883127    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:21:40.064967 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:40.064930    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"]
Nov 03 17:21:40.111900 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:40.111870    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m9ls"]
Nov 03 17:21:40.549448 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:21:40.549415    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c364a8d-dfea-461d-9ba9-3ad505bc1a9b" path="/var/lib/kubelet/pods/9c364a8d-dfea-461d-9ba9-3ad505bc1a9b/volumes"
Nov 03 17:22:07.882930 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:22:07.882894    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:22:37.882307 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:22:37.882271    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:22:40.693833 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:22:40.693796    5112 scope.go:122] "RemoveContainer" containerID="3c991dca2fc86bf9199ae9f65db9f3ecbcfa5e7ab530b315fe9a3c2d3aa19f95"
Nov 03 17:22:40.702933 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:22:40.702911    5112 scope.go:122] "RemoveContainer" containerID="84b4e92691232fa4703b9690485383226962b4633d404b2ee78134d81d9e6c4a"
Nov 03 17:23:07.883185 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:23:07.883099    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:23:37.883214 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:23:37.883156    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:24:07.882959 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:24:07.882919    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:24:37.885529 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:24:37.885485    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:25:02.056630 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:02.056592    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"]
Nov 03 17:25:02.103758 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:02.103722    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8jc7t"]
Nov 03 17:25:02.549458 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:02.549422    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba87545-0ffb-4a00-be73-a8d764f83385" path="/var/lib/kubelet/pods/bba87545-0ffb-4a00-be73-a8d764f83385/volumes"
Nov 03 17:25:07.884579 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:25:07.884538    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:25:21.055436 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:21.055390    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"]
Nov 03 17:25:21.102446 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:21.102416    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-flkm6"]
Nov 03 17:25:22.548803 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:22.548768    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65d54f3-ba19-47e4-bd17-c84b911335f7" path="/var/lib/kubelet/pods/a65d54f3-ba19-47e4-bd17-c84b911335f7/volumes"
Nov 03 17:25:37.882584 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:25:37.882545    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:25:39.528376 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:39.528346    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:25:39.538929 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:39.538904    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:25:39.654427 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:39.654399    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:25:39.668743 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:25:39.668712    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:26:00.071007 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:00.070919    5112 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"]
Nov 03 17:26:00.117909 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:00.117868    5112 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffe287a6-rrdvg"]
Nov 03 17:26:00.548874 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:00.548837    5112 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f73755-c032-44b2-bb80-baa61eaa2ce9" path="/var/lib/kubelet/pods/01f73755-c032-44b2-bb80-baa61eaa2ce9/volumes"
Nov 03 17:26:01.600359 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:01.600320    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66ff8f5b47-kph6d_3f4c81cc-7834-49dd-b89d-5b4b3189e013/authorino/0.log"
Nov 03 17:26:07.882513 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:26:07.882465    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:26:22.080241 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:22.080187    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-778b77c46d-wg5sn_3f58454f-bc25-4bae-a8ff-aa43c025d537/manager/0.log"
Nov 03 17:26:22.450542 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:22.450512    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hvcc9_b00b85c8-024d-4cba-be13-1963655418ed/manager/0.log"
Nov 03 17:26:22.724265 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:22.724165    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tm5k_f7a492f5-71b4-4348-886d-72bafe60d156/manager/0.log"
Nov 03 17:26:23.498299 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:23.498269    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b46f6b789-vpnvw_75918970-083d-41a0-aa18-05a395d1f934/manager/0.log"
Nov 03 17:26:24.153926 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:24.153897    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2r7rn_d3a3c2ca-0044-4f06-9178-8baa1a312c2c/manager/0.log"
Nov 03 17:26:25.212669 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:25.212639    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-855b85cfb4-bglrh_1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d/maas-api/0.log"
Nov 03 17:26:26.641837 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:26.641761    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-sldd2_1937961f-b133-4de5-b7cd-7011593da92a/postgres/0.log"
Nov 03 17:26:28.131294 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:28.131265    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66ff8f5b47-kph6d_3f4c81cc-7834-49dd-b89d-5b4b3189e013/authorino/0.log"
Nov 03 17:26:28.592731 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:28.592683    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tggc5_d08ca72c-70da-42cd-8ea9-b44a5f078c2d/manager/0.log"
Nov 03 17:26:29.111693 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:29.111666    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6j829_3fbdbf34-dc5c-4f9c-87a1-c6b391294237/kuadrant-console-plugin/0.log"
Nov 03 17:26:29.401380 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:29.401348    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-thzcr_e16d16c1-4f21-4007-9dbf-c66a4fd290a0/manager/0.log"
Nov 03 17:26:30.005765 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:30.005734    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5d2fl_f78e2948-1cd5-4098-84d4-ae66c985fdba/manager/0.log"
Nov 03 17:26:30.817618 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:30.817588    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886f59wpd_7feef304-fd59-4aa7-adb6-adbeacd388b6/istio-proxy/0.log"
Nov 03 17:26:37.884368 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:26:37.884328    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:26:40.754185 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:40.754151    5112 scope.go:122] "RemoveContainer" containerID="15ad08322cba56b7172d406a62572403ccd5ae6d5a88a50a2367cfee0eeed39f"
Nov 03 17:26:40.763453 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:40.763429    5112 scope.go:122] "RemoveContainer" containerID="db7c5084c7f877185a068b853644e82f49fdbb66aa3acbf384a1a82f6f21a779"
Nov 03 17:26:41.852805 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:41.852769    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-2jzfr_c132a838-d7ba-4445-a3e7-f5e4955d4688/istio-proxy/0.log"
Nov 03 17:26:42.155905 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:42.155872    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-2wnwx_106b94ab-7184-4c6b-b0e4-3bf6a4b763a0/istio-proxy/0.log"
Nov 03 17:26:42.384556 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:42.384524    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-fgmxm_ffae2951-6be1-4108-9570-62531dccc289/istio-proxy/0.log"
Nov 03 17:26:44.665888 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:44.665851    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-b9ppx_9b88bbf9-f49b-4e54-9eea-900cc0e5e078/payload-pre-processing/0.log"
Nov 03 17:26:44.892490 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:44.892462    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5j5sw_8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4/payload-processing/0.log"
Nov 03 17:26:45.124671 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:45.124593    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-689b56979-4v24f_36fe958f-eb20-4410-9e52-2a82273fce44/router/0.log"
Nov 03 17:26:46.015837 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:46.015805    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw_66361ba9-7dcb-49a9-b539-d6a40a5be289/main/0.log"
Nov 03 17:26:46.772587 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:46.772549    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4v77c_32d8a9c3-50b8-4487-951a-0963519e47b5/main/0.log"
Nov 03 17:26:47.033539 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:47.033464    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z_465a697d-a817-4a5d-af72-11a56526571f/main/0.log"
Nov 03 17:26:47.522624 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:47.522596    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn_26d95fe7-b2ba-4872-9587-647d5488e5a2/main/0.log"
Nov 03 17:26:58.686366 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:26:58.686323    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66ff8f5b47-kph6d_3f4c81cc-7834-49dd-b89d-5b4b3189e013/authorino/0.log"
Nov 03 17:27:07.883594 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:27:07.883557    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:27:27.667119 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:27.667041    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-778b77c46d-wg5sn_3f58454f-bc25-4bae-a8ff-aa43c025d537/manager/0.log"
Nov 03 17:27:28.033644 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:28.033558    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hvcc9_b00b85c8-024d-4cba-be13-1963655418ed/manager/0.log"
Nov 03 17:27:37.882519 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:27:37.882484    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:27:38.260280 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:38.260245    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5tm5k_f7a492f5-71b4-4348-886d-72bafe60d156/manager/0.log"
Nov 03 17:27:39.076949 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:39.076917    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b46f6b789-vpnvw_75918970-083d-41a0-aa18-05a395d1f934/manager/0.log"
Nov 03 17:27:39.678069 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:39.678039    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2r7rn_d3a3c2ca-0044-4f06-9178-8baa1a312c2c/manager/0.log"
Nov 03 17:27:40.554447 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:40.554416    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-855b85cfb4-bglrh_1ae9dc0f-861f-4546-a2b0-f0ffaf1de33d/maas-api/0.log"
Nov 03 17:27:40.789739 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:40.789703    5112 scope.go:122] "RemoveContainer" containerID="3f46244399841323cb8ee128c1401e8659aa29ab85430a0a06370811207021d9"
Nov 03 17:27:41.849385 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:41.849355    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-sldd2_1937961f-b133-4de5-b7cd-7011593da92a/postgres/0.log"
Nov 03 17:27:43.024748 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:43.024706    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66ff8f5b47-kph6d_3f4c81cc-7834-49dd-b89d-5b4b3189e013/authorino/0.log"
Nov 03 17:27:43.497682 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:43.497655    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tggc5_d08ca72c-70da-42cd-8ea9-b44a5f078c2d/manager/0.log"
Nov 03 17:27:43.947062 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:43.947036    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6j829_3fbdbf34-dc5c-4f9c-87a1-c6b391294237/kuadrant-console-plugin/0.log"
Nov 03 17:27:44.168837 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:44.168809    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-thzcr_e16d16c1-4f21-4007-9dbf-c66a4fd290a0/manager/0.log"
Nov 03 17:27:44.684654 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:44.684628    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5d2fl_f78e2948-1cd5-4098-84d4-ae66c985fdba/manager/0.log"
Nov 03 17:27:45.267475 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:45.267437    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886f59wpd_7feef304-fd59-4aa7-adb6-adbeacd388b6/istio-proxy/0.log"
Nov 03 17:27:46.310098 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:46.310069    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-2jzfr_c132a838-d7ba-4445-a3e7-f5e4955d4688/istio-proxy/0.log"
Nov 03 17:27:46.633847 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:46.633745    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-2wnwx_106b94ab-7184-4c6b-b0e4-3bf6a4b763a0/istio-proxy/0.log"
Nov 03 17:27:46.916521 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:46.916490    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-fgmxm_ffae2951-6be1-4108-9570-62531dccc289/istio-proxy/0.log"
Nov 03 17:27:49.373635 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:49.373604    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-b9ppx_9b88bbf9-f49b-4e54-9eea-900cc0e5e078/payload-pre-processing/0.log"
Nov 03 17:27:49.592291 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:49.592260    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5j5sw_8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4/payload-processing/0.log"
Nov 03 17:27:49.817612 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:49.817579    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-689b56979-4v24f_36fe958f-eb20-4410-9e52-2a82273fce44/router/0.log"
Nov 03 17:27:50.462280 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:50.462253    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8kgbw_66361ba9-7dcb-49a9-b539-d6a40a5be289/main/0.log"
Nov 03 17:27:51.176638 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:51.176604    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4v77c_32d8a9c3-50b8-4487-951a-0963519e47b5/main/0.log"
Nov 03 17:27:51.454812 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:27:51.454731    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ml68z_465a697d-a817-4a5d-af72-11a56526571f/main/0.log"
Nov 03 17:28:01.941759 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:28:01.941729    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cpkbn_26d95fe7-b2ba-4872-9587-647d5488e5a2/main/0.log"
Nov 03 17:28:07.882348 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:28:07.882306    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:28:37.882518 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:28:37.882480    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:28:50.539905 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:28:50.539816    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b46f6b789-vpnvw_75918970-083d-41a0-aa18-05a395d1f934/manager/0.log"
Nov 03 17:28:51.112375 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:28:51.112343    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b46f6b789-vpnvw_75918970-083d-41a0-aa18-05a395d1f934/manager/0.log"
Nov 03 17:28:51.845877 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:28:51.845850    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5j5sw_8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4/payload-processing/0.log"
Nov 03 17:28:52.066911 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:28:52.066886    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5j5sw_8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4/payload-processing/0.log"
Nov 03 17:29:07.882213 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:29:07.882162    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:29:37.883411 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:29:37.883373    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:30:07.882914 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:30:07.882868    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:30:07.950047 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:07.950012    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b46f6b789-vpnvw_75918970-083d-41a0-aa18-05a395d1f934/manager/0.log"
Nov 03 17:30:08.527518 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:08.527491    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b46f6b789-vpnvw_75918970-083d-41a0-aa18-05a395d1f934/manager/0.log"
Nov 03 17:30:09.283834 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:09.283804    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5j5sw_8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4/payload-processing/0.log"
Nov 03 17:30:09.503463 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:09.503431    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-5j5sw_8a1a9cc3-468f-45fc-a0b9-bf9bd6ecc7d4/payload-processing/0.log"
Nov 03 17:30:20.441157 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:20.441084    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h7sv8_2b0366db-bfff-4d9c-af6e-37b79c084739/global-pull-secret-syncer/0.log"
Nov 03 17:30:21.660657 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:21.660630    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6l6l8_0dc284dd-fff4-439d-8732-7b78daf69085/connection-checker/0.log"
Nov 03 17:30:22.092594 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:22.092517    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cff6v_47333977-43b6-47af-ab13-b5dbee646be5/connection-checker/0.log"
Nov 03 17:30:22.534105 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:22.534075    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l9r4n_11ba431e-e81b-4dbf-9d65-445a66ff6e07/connection-checker/0.log"
Nov 03 17:30:24.074599 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:24.074551    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sg6b8_d5d79060-9c22-412b-b6df-576c7a2dd340/konnectivity-agent/0.log"
Nov 03 17:30:24.921169 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:24.921137    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-122.us-west-2.compute.internal_d9d1cb113f9a0ff39835d34e4e74ebc1/haproxy/0.log"
Nov 03 17:30:28.551724 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:28.551697    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66ff8f5b47-kph6d_3f4c81cc-7834-49dd-b89d-5b4b3189e013/authorino/0.log"
Nov 03 17:30:28.888263 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:28.888155    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tggc5_d08ca72c-70da-42cd-8ea9-b44a5f078c2d/manager/0.log"
Nov 03 17:30:29.328103 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.328066    5112 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"]
Nov 03 17:30:29.328587 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.328573    5112 cpu_manager.go:414] "containerMap: removing container" podUID="eb8f24db-ec68-4061-9a03-c32e8c79b046" containerName="maas-api"
Nov 03 17:30:29.328636 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.328589    5112 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8f24db-ec68-4061-9a03-c32e8c79b046" containerName="maas-api"
Nov 03 17:30:29.328710 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.328700    5112 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8f24db-ec68-4061-9a03-c32e8c79b046" containerName="maas-api"
Nov 03 17:30:29.331992 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.331970    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.383411 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.383375    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwpzh\"/\"kube-root-ca.crt\""
Nov 03 17:30:29.427551 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.427518    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-sys\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.427719 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.427661    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf856\" (UniqueName: \"kubernetes.io/projected/0ebae853-0bfa-4999-bd38-49e61b03c89c-kube-api-access-qf856\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.427763 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.427715    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-lib-modules\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.427826 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.427808    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-podres\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.427868 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.427844    5112 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-proc\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.430165 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.430134    5112 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwpzh\"/\"openshift-service-ca.crt\""
Nov 03 17:30:29.430165 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.430160    5112 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mwpzh\"/\"default-dockercfg-c459v\""
Nov 03 17:30:29.432718 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.432690    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"]
Nov 03 17:30:29.528616 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528584    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-podres\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528616 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528627    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-proc\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528668    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-sys\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528718    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-podres\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528732    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf856\" (UniqueName: \"kubernetes.io/projected/0ebae853-0bfa-4999-bd38-49e61b03c89c-kube-api-access-qf856\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528774    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-proc\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528785    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-sys\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528781    5112 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-lib-modules\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.528879 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.528843    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ebae853-0bfa-4999-bd38-49e61b03c89c-lib-modules\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.585323 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.585232    5112 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf856\" (UniqueName: \"kubernetes.io/projected/0ebae853-0bfa-4999-bd38-49e61b03c89c-kube-api-access-qf856\") pod \"perf-node-gather-daemonset-79z9f\" (UID: \"0ebae853-0bfa-4999-bd38-49e61b03c89c\") " pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.644978 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.644936    5112 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:29.772164 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:29.772131    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6j829_3fbdbf34-dc5c-4f9c-87a1-c6b391294237/kuadrant-console-plugin/0.log"
Nov 03 17:30:30.016041 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:30.016011    5112 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"]
Nov 03 17:30:30.017542 ip-10-0-137-122 kubenswrapper[5112]: W1103 17:30:30.017513    5112 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0ebae853_0bfa_4999_bd38_49e61b03c89c.slice/crio-8cf617b5a6830bc51e85cecce6583385227577b5aadb54bd24919ce28694d5fd WatchSource:0}: Error finding container 8cf617b5a6830bc51e85cecce6583385227577b5aadb54bd24919ce28694d5fd: Status 404 returned error can't find the container with id 8cf617b5a6830bc51e85cecce6583385227577b5aadb54bd24919ce28694d5fd
Nov 03 17:30:30.240903 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:30.240865    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-thzcr_e16d16c1-4f21-4007-9dbf-c66a4fd290a0/manager/0.log"
Nov 03 17:30:30.280346 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:30.280270    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f" event={"ID":"0ebae853-0bfa-4999-bd38-49e61b03c89c","Type":"ContainerStarted","Data":"e3987cd34e322a4e9b9de97e24e7a9668c96cfc4dd48bc59b15659f328f4194f"}
Nov 03 17:30:30.280346 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:30.280335    5112 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f" event={"ID":"0ebae853-0bfa-4999-bd38-49e61b03c89c","Type":"ContainerStarted","Data":"8cf617b5a6830bc51e85cecce6583385227577b5aadb54bd24919ce28694d5fd"}
Nov 03 17:30:30.280525 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:30.280393    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:30.392480 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:30.392427    5112 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f" podStartSLOduration=1.392412146 podStartE2EDuration="1.392412146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:30.391083198 +0000 UTC m=+4791.970387053" watchObservedRunningTime="2026-11-03 17:30:30.392412146 +0000 UTC m=+4791.971716001"
Nov 03 17:30:31.133729 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:31.133699    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5d2fl_f78e2948-1cd5-4098-84d4-ae66c985fdba/manager/0.log"
Nov 03 17:30:36.299429 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:36.299394    5112 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mwpzh/perf-node-gather-daemonset-79z9f"
Nov 03 17:30:36.920479 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:36.920447    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-jnx8b_ef6cea18-3a39-4c4b-8fba-0284ca6f2e98/cluster-monitoring-operator/0.log"
Nov 03 17:30:37.885980 ip-10-0-137-122 kubenswrapper[5112]: E1103 17:30:37.885943    5112 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8szwq" containerName="extractor"
Nov 03 17:30:39.567037 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:39.567004    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:30:39.575544 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:39.575518    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9q49_2a8de62e-d0ca-4ee7-980a-ae48558a87ee/node-exporter/0.log"
Nov 03 17:30:39.577616 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:39.577597    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:30:39.693973 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:39.693948    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdxst_f6e8d4ca-17c7-4c23-bc88-934fa6824168/extract/0.log"
Nov 03 17:30:39.703549 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:39.703522    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l2th2_290e7793-2a71-4dfb-9375-be6c7d26d7bb/console-operator/1.log"
Nov 03 17:30:40.017360 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:40.017319    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9q49_2a8de62e-d0ca-4ee7-980a-ae48558a87ee/kube-rbac-proxy/0.log"
Nov 03 17:30:40.487185 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:40.487157    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9q49_2a8de62e-d0ca-4ee7-980a-ae48558a87ee/init-textfile/0.log"
Nov 03 17:30:48.881393 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:48.881361    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-s557v_f2972ac4-c9e1-45be-ba6d-0b5199d445a7/prometheus-operator-admission-webhook/0.log"
Nov 03 17:30:51.940170 ip-10-0-137-122 kubenswrapper[5112]: I1103 17:30:51.940126    5112 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886f59wpd_7feef304-fd59-4aa7-adb6-adbeacd388b6/istio-proxy/0.log"